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ABSTRACT 


EVALUATING  THE  EFFECTIVENESS  OF  INFORMATION  CENTRES  AND  SERVICES 

This  Lecture  Series  presents  the  current  state-of-the-art  of  evaluating  information  centres  and  services  both  from  a 
theoretical  and  practical  perspective.  Case  studies  are  used  to  demonstrate  successful  evaluation  projects  from  different 
countries  and  to  focus  on  specific  evaluation  methods  and  measures.  In  addition  to  providing  a  thorough  basis  for  determining 
the  most  appropriate  evaluation  methods  and  measures  for  a  given  situation  and  potential  problems  that  can  arise  in  using 
them,  the  presentations  include  discussions  of  what  to  do  once  the  evaluation  is  done.  It  proposes  various  strategies  for  altering 
the  centres  and  services  so  that  effectiveness  is  improved;  in  so  doing,  it  covers  political,  organizational  and  practical  aspects  of 
evaluation  projects  so  that  evaluation  is  seen  from  its  broader  contextual  location. 


Ce  Cycle  de  Conferences  presente  I'etat  actuel  des  connaissances  dans  le  domaine  de  revaluation  des  centres  et  des 
services  d’information,  tant  sur  le  plan  theorique  que  pratique.  Des  cas  pratiques  sont  examines,  afin  d’illustrer  un  certain 
nombre  de  projets  d’evaluation  qui  ont  etc  entrepris  avec  succes  par  diSerents  pays,  en  mettant  I'accent  sur  des  methodes  et  des 
actions  d'evaluation  specifiques. 

Ainsi,  les  presentations  constituent  non  seulement  des  bases  solides  pour  la  definition  des  methodes  et  des  actions 
d’evaluation  les  plus  appropriees  par  rapport  a  une  situation  donnee  et  des  problemes  susceptibles  de  se  poser  lors  de  leur  mise 
en  oeuvre,  mais  aussi  la  definition  des  orientations  cnncemant  les  mesures  i  prendre  d»  I'achevement  de  revaluation. 

Differentes  strategies  sont  proposees  pour  la  modification  des  centres  et  des  services  dans  le  sens  d'une  meilleur  efficacite, 
en  soulignant  les  aspects  politiques,  d’otgaiusation  et  pratiques  des  projets  d'evaluation,  vu  sous  I’angle  d'un  contexte  elargi. 
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This  paper  describes  an  approach  to  evaluating  the  use,  usefulness  and  value  of 
published/recorded  information  on  the  work  of  its  users.  The  reading  patterns  of 
professionals  the  application  of  information  derived  from  reading  is  discussed  in 
detail.  Several  different  indicators  of  productivity  of  professionals  are  developed  and 
correlated  with  the  amount  of  reading  that  is  done.  Finally,  several  different 
iwrspectives  on  the  value  of  information  centres  to  tto  professionals  they  serve  are 
presented. 


In  the  paper  "A  Framework  for  Evaluating  the  Effectiveness  of  Information  Centres 
and  Services,  "  we  indicated  that  information  is  one  of  several  resources  used  by 
professionals  to  perform  their  work.  In  this  paper  we  provide  evidence  of  the  extent  to 
which  information  affects  input  costs,  output  performance  and  productivity  of 
professionals.  In  subsequent  papers  we  show  that  information  centres  contribrte 
substantially  to  the  use,  usefulness  and  value  of  information  used  by  professionals. 
Furthermore,  the  performance  of  information  centres'  operations  and  services  determine 
to  a  great  extent  the  contribution  that  is  made  to  the  organization  served. 

Professionals  such  as  scientists,  engineers,  lawyers,  medical  practitioners, 
managers,  administrators,  and  so  on,  spend  about  60  percent  of  their  time  communicating. 
This  communication  can  take  the  form  of  interpersonal  communication,  such  as  talking  or 
listening  to  someone,  or  participation  in  presentations  to  groups.  It  can  also  involve 
reading  or  writing.  Typically,  about  70  percent  of  the  time  spent  communicating 
involves  interpersonal  exchange  of  information,  and  30  per'^ent  reading  and  writing.  The 
rest  of  the  time  of  professionals  is  spent  actually  performing  primary  work  activities 
such  as  thinking,  conducting  primary  or  laboratory  research,  preparing  engineering 
designs  or  drawings,  diagnosing  or  treating  patients,  preparing  legal  strategies, 
planning  and  budgeting,  and  so  on. 

We  find  that  all  professionals,  regardless  of  work  role  or  educational  background, 
spend  about  the  eanve  amount  of  time  communicating?  however,  the  way  they  communicate 
varies  substantially.  For  example,  scientists  typically  read  and  write  more  than  other 
professionals,  and  the  materials  they  read  tend  to  be  more  from  scholarly  journals  and 
books  than  from  trade  journals  and  non^scholarly  books;  whereas  administrators  tend  to 
read  much  more  frcxn  trade  journals  than  txcaa  any  other  type  of  document.  Administrators 
also  spend  much  less  time  reading  than  other  professionals.  For  example,  administrators 
average  about  100  hours  per  year  reading  compared  with  about  330  for  researchers  and 
over  500  hours  for  lawyers.  However,  they  do  spend  more  time  in  interpersonal 
c  omnun  1  c  a  t  i  on . 

In  order  to  perform  their  work,  professionals  use  various  resources  such  as  their 
own  time,  computing  equijxnent,  support  staff,  advice  from  consultants  or  colleagues,  and 
information  found  in  documents.  Presumably  each  resource  makes  a  contribution  to  a  work 
activity's  input  cost  and  output  performance.  Usually,  but  not  always,  better  resources 
cost  more  but  should  achieve  greater  output  performance  in  terms  of  quantities  produced, 
quality,  timeliness,  etc.  Measuring  output  performance  is  difficult,  but  useful 
Indicators  of  such  performance  can  be  observed.  For  example,  output  quantities  can  be 
measured  in  terms  of  formal  records  or  reports  written  (e.g.  technical  reports,  lib 
notes,  etc.),  written  proposals  or  plans,  presentations  made  (e.g.,  about  research 
results,  proposals  or  plans,  etc.),  and  so  on.  Productivity  of  professionals  is  the 
relationship  between  output  performance  and  input  costs  (e.g.,  amount  of  research 
recorded  divided  by  scientists*  hours  spent  performing  research). 

In  our  studies,  we  have  carefully  examined  one  resource  used  by  professionals: 
information  found  in  documents.  For  a  range  of  work  activities,  we  have  observed 
professionals'  assessments  of  the  relative  importance  of  information  and  other 
resources.  We  have  also  established  indicators  of  the  effect  of  information  from 
documents  on  input  costs  and  output  performance  (in  terms  of  output  quantities,  quality 
and  timeliness).  Finally,  we  provide  indicators  of  the  effect  of  information  on 
productivity,  in  terms  of  several  types  of  output  quantities  divided  by  relevant  number 
of  hours  of  professional  time  spent  producing  them. 

Input  costs  associated  with  information  found  in  documents  is  measured  by  the  costs 
of  obtaining  documents  and  of  actually  reading  them.  We  find  that  the  reading  costs 
tend  to  dominate  the  costs  of  obtaining  the  documents  (by  a  factor  of  about  five  to 
one) .  Although  the  amount  and  type  of  reading  varies  among  professionals  performing 
different  work  roles,  professionals  from  all  work  roles  read  a  great  deal  and  spend 
substantial  time  on  this  reading.  For  example,  the  average  annual  amount  of  reading  and 
average  annual  time  spent  reading  per  professional  in  various  work  roles  are  observed  as 
follows: 


Manage-  Adminis-  Operations 


Annual  Information 

ment 

tration 

and  Other 

R&D 

Medical 

Legal 

Number  of  Items  Read* 

260 

110 

180 

240 

240 

370 

Time  Spent  Reading 

230 

100 

200 

330 

270 

510 

Professionals'  time  is  a  scarce  resource  and,  therefore,  if  they  choose  to  spend  100  to 
500  hours  reading  each  year  (depending  on  work  role),  the  knowledge  gained  must  be  of 
substantial  usefulness  and  value  to  them*  In  fact,  we  have  gathered  a  great  deal  of 
evidence  that  this  is  true.  In-depth  studies  in  twelve  companies  or  government 
agencies,  provide  evidence  that  information  is  indeed  important  and  may  have  an 
appreciable  effect  on  professionals*  work. 

The  principal  purposes  for  which  reading  is  done  are  to  accomplish  the 
professionals'  primary  work  such  as  research  and.  development,  management, 
administration,  and  so  on.  For  example,  about  60  percent  of  the  reading  of  journal 
articles  is  for  the  purpose  of  accomplishing  these  primary  work  activities.  Some 
reading  (54)  is  done  for  advising  others,  making  presentations  or  educating  others;  some 
reading  (5%)  is  for  writing  reports,  proposals  or  plans,  and  publications.  About  30 
percent  of  the  reading  of  journal  articles  is  for  professional  development.  These 
purposes  of  reading  vary  substantially  depending  on  work  roles  of  professionals.  For 
example,  only  about  one-fifth  of  the  journal  reading  for  research  and  development 
professionals  is  for  professional  development;  whereas,  nearly  one-half  of  the  journal 
reading  of  administrators  is  for  this  purpose.  Administrators  are  much  more  likely  than 
other  professionals  to  read  journals  for  the  purpose  of  consulting  or  giving  substantive 
advice  to  others. 

Professionals  employ  a  number  of  resources  in  doing  their  work.  Such  resources 
include  computers,  terminals,  and  other  equipment  (such  as  laboratory  instrumentation); 
information  found  in  documents;  substantive  advice  from  colleagues  or  consultants; 
support  staff,  such  as  secretaries,  technicians,  etc.;  and  information  staff,  such  as 
librarians,  information  specialists,  etc.  Obviously,  the  most  important  resource  of  all 
is  the  time  and  effort  of  professionals  themselves.  The  relative  importance  of  the 
remaining  resources  varies  among  professionals  and  the  work  activities  for  which  the 
resources  are  employed.  For  example,  instrumentation  is  the  most  important  resource  to 
engineers  in  doing  primary  research;  but  this  resource  is  of  little  use  to 
administrators  because  they  rarely  perform  the  activity.  Information  found  in  documents 
is  found  to  be  important  to  all  types  of  work  activities  and  it  is  the  most  important 
resource  for  such  activities  as  background  research  and  other  non-primary  R6D 
activities;  writing  proposals  or  plans,  reports  and  formal  publications;  educating  and 
training  others;  consulting  and  advising  others;  and  professional  development. 

We  also  established  several  indicators  of  the  beneficial  consequences  of  reading 
books,  journals  and  technical  reports.  One  of  the  most  significant  consequences  of 
reading  these  materials  is  that  considerable  dollar  savings  are  achieved  by  avoiding 
having  to  do  certain  work  at  all,  modifying  the  work  or  stopping  an  unproductive  line  of 
work.  Again,  the  proportion  of  readings  which  result  in  such  savings  and  the  estimated 
amount  of  such  savings  vary  substantially  among  professionals  and  their  work  roles. 
These  consequences  also  vary  by  the  types  of  materials  read.  However,  such  savings 
across  all  professionals  and  all  reading  in  an  organization  amount  to  a  dollar  value 
that  is  approximately  ten  times  that  of  the  cost  of  identifying,  accessing  and  actually 
reading  the  documents.  Thus,  the  return  on  investment  in  information  found  in  documents 
is  ten  to  one  (using  this  criterion)**. 

For  their  most  recent  readings  of  journals,  books  and  technical  reports  we  asked 
professionals  to  indicate  the  effect  of  the  information  read  on  quality  and  timeliness 
of  the  work  activity  for  which  the  information  was  read.  We  found  that  a  large 
proportion  of  readings  affect  the  quality  of  the  primary  activity  for  which  the  reading 
was  done  (44%  of  journal,  61%  of  book  and  65%  of  report  readings).  Similarly,  reading 
was  also  said  to  improve  the  timeliness  of  the  activity,  but  not  for  nearly  as  many 
readings  as  for  improved  quality.  Timeliness  improvements  were  said  to  occur  for  19 
percent  of  journal  readings,  24  percent  of  book  readings  and  22  percent  of  report 
readings.  Clearly,  these  results  are  merely  indicators  or  evidence  of  the  extent  to 
which  Information  affects  the  output  performance  of  professionals;  nevertheless,  all 
such  indicators  have  demonstrated  the  use,  usefulness  and  value  of  information  found  in 
documents.*** 

*  Reading  is  defined  as  going  beyond  the  contents  page,  tit'e  page  and  abstract  to  the 
body  of  the  document.  Type  of  items  read  include  journal  articles  (trade  and 
scholarly),  books,  reports,  patents,  etc. 

**  We  have  observed  this  relative  return  on  information  on  three  national  surveys  of 
scientists  and  engineers  as  well  as  twelve  in-depth  studies  performed  in  companies 
and  government  agencies. 

***  Other  benef'^ts  of  reading  that  were  frequently  cited  by  professionals  include 
reinforcing  hypotheses  or  confidence  in  work,  initiated  new  ideas  for  work,  helped 
guide  future  work,  broadened  or  narrowed  options  concerning  work,  provided  needed 
Intelligence  about  competitors  and  provided  information  for  lectures,  seminars,  etc. 


One  important  indicator  of  the  value  of  information  found  in  documents  involves 
effect  of  reading  on  productivity  of  professionals.  Productivity  can  be  measured  in 
terms  of  such  output  quantities  as  nuo^er  of  formal  records  (of  research,  management, 
etc.)  written,  nuztdser  of  formal  publications  written,  number  of  proposals  or  research 
plans  prepared,  number  of  formal  oral  presentations  made,  and  number  of  times  in  which 
professionals  are  consulted  or  in  which  they  give  substantive  advice.  In  all  instances, 
productivity  was  found  to  be  statistically  correlated  with  amount  of  reading  of 
professionals.  For  example,  one  indicator  of  productivity  is  the  number  of  formal 
records  of  research  and  technical  work,  divided  by  time  actually  spent  doing  primary  and 
secondary  research  (and  development)  and  in  preparing  such  records.  For  252 
professionals  we  observed  the  following  relationship  between  amount  of  reading  and  level 
of  productivity*. 


Productivity 

LOW  High  Total 


1 

High  1 

1 

50 

!  77 

— 

1 

1 

127 

Amount  of 
Reading 

1 

1 

Low  1 

84 

41 

1 

1 

1 

125 

1 

Total  1 

134 

j  118 

1 

1 

1 

252 

It  seems  clear  that  professionals  who  read  a  great  deal  are  more  likely  to  also  have 
high  productivity  (77  of  127)  and,  conversely,  those  who  do  not  read  much  are  less 
likely  to  have  high  productivity  (84  of  125)**.  We  emphasize,  however,  that  we  do  not 
know  for  sure  whether  high  productivity  is  directly  attributable  to  frequent  reading. 
We  only  know  that  those  who  read  more  (or  less)  tend  to  have  higher  (or  lower) 
productivity  and  that  the  observed  relationship  is  unlikely  to  have  happened  by  chance. 
We  observed  similar  relationships  of  amount  of  reading  and  productivity  for  four  other 
indicators  of  productivity. 

Finally,  we  looked  at  several  indicators  of  the  contribution  that  professionals 
make  to  their  organizations  to  determine  whether  those  who  make  greatest  contributions 
read  more  or  less  than  others.  One  indicator  of  contribution  is  salary.  Presumably, 
those  who  contribute  more  make  a  larger  salary.  When  we  pair  professionals  within  work 
role,  degrees  and  field,  and  years  of  experience,  we  find  that  those  who  read  more  are 
paid  more  on  the  average.  In  some  companies  we  were  given  the  names  of  professionals 
who  had  received  achievement  awards,  technical  awards,  patent  recognition,  and  so  on. 
Another  measure  of  achievement  is  whether  a  professional  is  asked  to  serve  on  a  project 
or  problem-solving  teams  or  special  committees.  In  both  instances  (i.e.,  award  winners 
and  those  chosen  to  serve  on  special  teams)  the  recognized  achievers  read  significantly 
more  than  others.  Such  findings  are  not  new.  Similar  evidence  was  presented  by  Lufkin 
who  found  in  u  study  of  engineers  "that  the  people  who  have  b^en  singled  out  for 
excellence,  whether  by  promotion,  or  by  publication,  or  by  special  recognition  for 
creativity,  all  read  a  great  deal  more  than  the  average". 

We  feel  that  there  is  abundant  evidence  or  indicators  that  information  from 
documents  contribute  significantly  to  professionals’  work,  thereby  giving  heavy  readers 
an  information  edge.  Such  information: 

o  is  used  for  important  work  activities; 

o  has  substantial  value  (or  professionals  would  not  devote  their  scarce  Mme 
reading) ; 

o  is  said  to  be  an  important  resource  for  most  work  activities  and  it  is  the  most 
important  resource  (compared  to  other  resources)  for  several  work  activities; 

o  results  in  large  dollar  savings  in  staff  time  and  equipment; 

o  results  in  higher  quality  and  more  timely  work;  and 

o  amount  of  reading  is  correlated  with  productivity  measured  in  several  ways. 

This  evidence  quantifies  the  information  advantage  that  can  be  gained  by 
professionals  through  reading.  Below  we  discuss  how  information  centres  increase  this 
information  advantage. 


*  Note  that  the  level  of  high  (and  low)  productivity  and  high  (and  low)  amount  of 
reading  varies  among  organizations. 

**  A  statistical  test  (Chi-square)  rejects  the  hypothesis  (at  «  0.01)  that 
productivity  and  reading  are  independent  (i.e.,  no  relationship  exists). 
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THB  VALOB  OP  XBPORIATION  CBRntBS 

There  are  three  levela  that  are  considered  in  assessing  the  contribution  that 
infomation  centres  make  to  the  value  of  information.  Centres  are  not  inexpensive  to 
(derate.  We  find  from  national  surveys  that  cofttpanies  and  government  agencies  typically 
spend  from  $500  to  $1,500  per  professional  (user  or  not)  on  information  centres.  On 
behalf  of  the  organization,  the  centres  buy  journals,  books  and  other  materials. 
Acquiring,  processing,  maintaining  and  distributing  these  materials  in  a  timely  way  is 
very  labor-intensive.  Furthermore,  they  provide  a  range  of  other  services  such  as 
reference,  online  searching,  translation,  and  so  on.  In  the  eyes  of  management,  there 
must  be  a  clear  demonstration  of  favourable  return  on  this  investment. 

The  first  question  that  comes  to  mind  is  whether  the  price  paid  for  information 
materials  has  a  concomitant  value.  One  envious  way  of  assessing  this  value  is  from  the 
perspective  of  the  readers.  What  are  they  willing  to  pay  for  this  information?  We  can 
readily  measure  what  they  do  pay,  recognizing  that  they  might  pay  more  if  they  had  to. 
Time  of  professionals  is  a  scarce  resource.  Professionals  must  decide  how  to  utilize 
their  time  in  order  to  be  most  productive.  Engineers,  scientists,  lawyers, 
administrators,  and  so  on,  devote  a  substantial  amount  of  their  time  to  accessing, 
reading  and  using  information  found  in  documents,  such  as  journals,  books,  technical 
reports,  patents,  and  so  on.  Their  decisions  to  use  their  scarce  time  for  information 
seeking  and  reading  are  a  strong  indication  of  the  value  they  place  on  this  information. 
The  total  time  (and  the  dollar  amount  represented  by  this  time)  that  is  spent  on 
information  provided  by  information  centres  is  an  indicator  of  value  of  the  information. 
A  large  number  of  studies  show  this  value  to  be  on'^the  order  of  four  to  five  times  that 
of  the  cost  to  information  centres  of  purchasing  and  providing  these  materials.  That 
is,  even  though  an  organization  may  pay  from  about  $500  to  $1,500  per  professionals  for 
services,  the  professionals  demonstrate  that  information  provided  by  information  centres 
has  a  value  of  about  $5,000.  Thus,  the  centres  clearly  are  providing  important 
information  and  services. 

Of  course,  the  Information  could  be  obtained  by  the  professionals  from  other 
sources.  They  could  stJbscribe  to  the  journals  or  purchase  the  books  and  technical 
reports  that  are  now  provided  by  information  centres.  Then  organizations  would  lose 
potential  savings  achieved  by  sharing  the  infrequently  read  materials,  and  professionals 
would  not  receive  information  and  services  in  a  timely  manner.  They  could  use  sources 
outside  of  their  organizations,  such  as  academic  libraries,  but  that  could  involve 
substantial  professional  time  required  to  identify,  locate  and  gain  access  to  materials 
found  in  these  other  locations.  They  could  order  materials  from  document  delivery 
services,  publishers  or  other  sources,  but  that  assumes  an  ability  to  identify  needed 
information  and  then  locate  where  it  is  and  acquire  it.  Furthermore,  if  all  company 
professionals  relied  on  academic,  public  or  other  libraries,  these  libraries  would  soon 
stop  making  their  collections  available  because  of  the  cost  and  possible  denial  of 
access  to  their  own  primary  users.  In  fact,  this  trend  has  already  begun  (e.g.,  at 
Princeton  University).  The  point  is  that  having  a  nearby  information  centre  within 
organizations  saves  professionals  considerable  time  and  money.  In  fact,  we  estimate 
these  savings  to  be  about  $4,800  per  professional  per  year.  Generally,  we  estimate 
these  savings  to  be  of  the  order  of  eight  times  the  cost  to  provide  these  materials 
through  an  in-house  information  centre.  Thus,  the  return  on  information  for  information 
centres  is  about  eight  to  one  when  viewed  in  this  way. 

About  two-thirds  of  the  cost  avoidance  achieved  by  the  information  centres  involves 
professional  time.  We  have  observed,  over  the  years,  that  professionals  tend  to  spend  a 
relatively  constant  proportion  of  their  time  in  information  seeking  and  reading.  The 
time  may  shift  from  accessing  to  reading  or  vice  versa,  but  the  total  seems  not  to  vary 
significantly.  With  this  in  mind,  we  developed  a  rationale  for  determining  what  would 
happen  if  professionals  had  to  rely  entirely  on  sources  other  than  information  centres. 
If  professionals  did  not  have  such  a  source  of  information,  one  could  safely  assume  that 
they  would  read  less  because  some  of  their  time  would  shift  from  reading  to  identifying, 
locating  and  acquiring  information  frcmi  alternative  sources,  (Note  that  not  all  the 
time  would  shift  to  information  seeking  artiv^tiesr  some  would  be  used  for  other 
work-related  activities).  Therefore,  benefits  would  not  be  derived  from  the  lv.,1 
readings.  Such  lost  benefits  include  savings  (in  time,  equipment,  etc.)  derived  from 
information  and  improvements  in  quality  of  work,  timeliness  of  work  output,  and  so  on. 
Such  lost  benefits  are  what  we  consider  to  be  the  highest  order  of  value  of  information 
centre  services.  This  value,  compared  with  the  cost  of  information  centres  is 
substantial.  The  lost  benefits  alone  are  found  to  be  of  the  order  of  ten  to  15  times 
the  cost  of  providing  the  services.  This  we  consider  to  be  the  ultimate  return  on 
information  for  information  centres. 


In  summary,  the  value  of  information  centre  services  can  be  assessed  from  three 
perspectives:  what  users  are  willing  to  pay  (in  terms  of  their  time  and  effort)  for 
information  provided  by  the  centre,  how  much  more  it  would  cost  them  to  use  alternative 
sources  for  obtaining  the  information,  and  what  benefits  (or  research  cost  avoidance) 
would  be  lost  if  the  centre  did  not  exist.  Rough  estimates  of  the  average  value  derived 
per  professional  for  these  three  perspectives  are  given  below. 


Value  Perspective 
Willingness  to  Pay 

Additional  Cost  to  Use  Alternative  Sources 
Lost  Benefits 


Value 
S  5,180 
$  4,760 
$27,000 


Return  on  Information 
8.6  to  1 
7.9  to  1 
45  to  1 
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Similar  results  have  been  reported  by  others  as  v.ell. 

Regardless  of  how  one  looks  at  it,  a  well  run  library  and  professional:?  who  know 
how  to  use  it  can  save  companies  and  government  agencies  an  enormous  amount  of  money. 
Also,  productivity  of  professionals,  quality  of  their  work  and  company  profits  or  leturn 
on  investment  in  prof essionals ’  time  are  likely  to  be  improved. 
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Loughborough  University  of  Technology 
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SUMARY 

Evaluating  the  effectiveness  of  organisations  and  of  the  services  they  offer  always  depends  on  assuming 
some  kind  of  model.  The  main  purpose  of  modelling  is  to  abstract  elements  from  real-life  activities  in 
such  a  way  that  these  activities  can  be  more  readily  understood,  and  so  better  organised  and  planned. 

Models  may  be  used  for  a  variety  of  purposes;  for  example,  to  pinpoint  which  parts  of  a  system  should  be 

sampled  to  provide  an  adequate  picture  of  the  whole.  The  drawback  of  all  models  is  that  they  reflect 
only  a  part  of  reality,  and  so  can  only  be  used  in  a  limited  context.  In  consequence,  models  must  be 
explicitly  formulated  and  their  limitations  understood  before  any  evaluation  exercise  is  attempted. 

The  model  helps  decide  what  performance  measures  can  usefully  be  made.  For  example,  a  qualitative  model 
may  not  be  appropriate  for  guiding  assessment  in  terms  of  quantitative  performance  measures.  Both  model 
and  measures  therefore  have  to  be  chosen  in  the  light  of  the  desired  outcome  from  the  evaluation.  The 
methodology  is.  in  turn,  primarily  determined  by  the  model  and  measures  chosen. 

This  talk  will  therefore  begin  by  looking  at  the  types  of  model  that  can  be  applied  to  evaluation.  It 

will  then  consider  what  sort  of  measures  might  be  desirable  in  different  context.  Finally,  the  question 
of  how  these  items  can  be  brought  together  in  practice  will  be  discussed. 

INTRODUCTION 

Assessing  how  well  an  Information  centre  or  service  is  running  entails  three  different  areas  of 

investigation.  The  first  relates  to  the  acquisition  of  information  by  the  centre  or  service,  and  what 

constraints  or  problems  are  involved  in  this.  The  second  concerns  the  centre  or  service,  itself,  and 
how  well  organised  it  is  to  handle  its  own  work.  The  final  area  involves  the  output  of  information  to 
users  and  how  well  their  requirements  are  satisfied.  Obviously,  all  three  areas  are  closely  linked:  a 
lack  of  Information  at  the  input  end,  for  example,  can  lead  to  dissatisfaction  by  users  at  the  output 
end.  It  is  a  short  step  from  accepting  this  to  seeing  an  information  centre  or  service  as  a  system  which 
can  be  analysed  in  terms  of  a  suitable  model.  Hence,  establishing  the  effectiveness  of  a  centre  or 
service  typically  begins  with  establishing  a  representative  picture,  or  model,  in  terms  of  which  its 
activities  can  be  studied. 

MODELLING 

The  word  'model'  is  used  in  a  number  of  different  ways.  For  our  purpose,  it  can  be  defined  as  meaning 
broadly  a  representation  of  something  else,  designed  as  an  aid  for  a  specific  purpose.  At  one  end  of 
this  spectrum  of  meanings,  the  representation  may  be  a  physical  entity.  For  example,  a  model  aeroplane 
can  be  placed  in  a  wind  tunnel  in  order  to  test  the  designer's  ideas  for  a  full-scale  aeroplane.  At  the 
other  end  of  the  spectrum,  the  representation  may  be  highly  theoretical.  The  motions  of  the  planets 

round  the  Sun  are  governed  by  a  very  complex  set  of  mathematical  equations  which  represent  the  current 

model  of  the  solar  system.  Despite  this  range  of  meaning,  models  have  a  number  of  characteristics  in 
common.  As  the  definition  emphasises,  they  are  almost  always  intended  to  serve  a  particular  purpose, 
rather  than  to  be  of  universal  application.  A  model  aeroplane  only  provides  information  in  the  wind 
tunnel  on  one  possible  configuration.  To  test  other  shapes,  new  models  must  be  built.  Each  model 
carries  wits  it  its  own  background  of  assumptions  and  presuppositions.  It  is  not  difficult  to  realise 
this  in  the  case  of  model  aeroplanes,  but  the  specificity  is  sometimes  less  obvious  for  the  type  of 
socio-economic  model  that  has  been  applied  to  studies  of  information  transfer.  In  fact,  the  latter  type 
of  model  is  often  much  simpler  in  appearance  than  the  complex  models  of  science  and  technology.  It  can 
give  the  generally  wrong  impression  that  socio-economic  models  are  straightforward  to  construct  and 
apply. 

A  typical  theoretical  model  might  be  constructed  in  the  following  way. 

(1)  The  process  to  be  examined  is  defined. 

(2)  The  variables  which  characterise  the  process  are  defined. 

(3)  The  relationships  between  these  variables  are  defined. 

(4)  The  parameters  (e.g.  Input/output  data)  for  the  process  are  determined. 

(5)  The  outcome  of  using  these  parameters  as  boundary  conditions  for  the  relationships  is  determined. 

(6)  This  outcome  Is  compared  with  the  observed  situation. 

(7)  An  attempt  is  made  to  see  whether  any  di Fferences  can  be  explained  as  due  to  errors  (in  the 
formulation  or  the  parameters). 

(8)  If  they  cannot,  a  new  or  modified  formulation  is  trlci. 

(9)  If  they  can,  the  formulation  is  identified  as  a  successful  model,  but  continues  to  be  evaluated  as 
new  data  appear. 

This  detailed  description  reflects  fairly  well  what  happens  in  the  development  of  a  macroscale 
econometric  model.  In  many  studies  of  information  transfer,  a  much  simpler  approach  is  used.  In  these, 
the  word  'model'  is  often  used  simply  for  a  mental  picture  of  the  processes  which  are  involved.  In 
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other  words,  it  only  implies  that  steps  (1)  and  (2)  in  the  list  have  been  taken.  This  can  be  illustrated 
by  the  following  example. 

Shannon's  work  on  information  transfer  immediately  after  the  Second  World  War  was  basically  concerned 
with  the  question  of  how  information,  mathematically  defined,  could  be  transmitted  down  a  telephone 
line.  It  was  concerned  with  the  quantitative  measurement  of  information  flow,  and  dealt  with  such 
entities  as  bits  per  event,  entropy  and  redundancy.  These  were  related  to  the  process  of  one  person 
speaking  into  a  telephone  and  another  person  at  the  other  end  receiving  the  message.  The  simple  model 
for  this  envisaged  by  Shannon  and  Weaver  (1)  represented  a  linear  message  flow  of  the  following  type. 


The  information  transfer  approach  developed  here  was  a  fully  fledged  scientific  model  involving  all  the 
steps  (1)  -  (9)  mentioned  above.  However,  the  breadth  of  its  significance  was  such  that  it  was  imported 
into  other  subject  areas,  gradually  shedding  some  of  its  characteristics  as  it  went.  Shannon's  original 
work  appeared  in  the  communications  engineering  literature  in  1948.  It  was  referred  to  in  psychology 
Journals  in  1949  and,  in  1950,  appeared  in  Journals  devoted  to  biology,  linguistics,  physics,  physiology 
and  sociology.  By  1955,  the  original  work  had  been  cited  in  at  least  17  different  subject  fields  in  the 
natural  and  social  sciences.  In  the  natural  sciences,  the  entire  model  was  of  interest,  but,  in  many  of 
the  social  sciences,  interest  centred  on  the  flow  diagram  for  information  transfer  which  was  often  taken 
over  by  itself.  Most  applications  of  the  Shannon-Weaver  model  today  to  socio-economic  discussions  are 
concerned  primarily  with  a  qualitative  picture  which  can  act  as  a  simple  model  to  guide  thinking.  Even 
if  some  of  the  refinements  of  the  original  treatment,  such  as  the  concept  of  'noise',  are  retained,  they 
are  typically  treated  in  a  qualitative,  rather  than  a  quantitative,  way. 

EXAMPLE  -  SYSTEM  DYNAMICS 

Not  all  methods  applied  to  socio-economic  questions  are  qualitative.  The  growth  of  computer  modelling 
has  made  it  possible  to  handle  quite  complex  information  transfer  activities  in  a  quantitative  way.  An 
example  is  provided  by  system  dynamics  modelling.  This  type  of  model  was  developed  by  J.  W.  Forrester 
at  MIT  in  the  1960s.  It  became  widely  known  as  a  result  of  its  use  by  the  Club  of  Rome  in  their 
'Limits  to  growth'  study.  Such  a  model  sees  information  activities  in  terms  of  an  information  flow 
system  containing  embedded  feedback  loops.  A  distinction  is  drawn  between  positive  and  negative  feed¬ 
back  loops.  The  former  destabilise  the  information  system,  whereas  the  latter  stabilise  it.  From  the 
computational  viewpoint,  a  system  dyn»ics  model  is  a  linked  set  of  differential  equations  which 
describe  the  continuous  changes  in  the  system.  The  system  behaviour  is  described  in  terms  of  the  rates 
at  which  the  variables  involved  change  with  time  and  the  corresponding  variations  in  their  levels.  An 
application  of  these  equations  provides  an  indication  of  how  decisions  about  the  operation  of  the  system 
will  affect  levels  and  rates  of  change. 

To  take  a  specific  example,  if  a  new  online  service  is  to  be  introduced,  what  can  be  said  about  the 
likely  rate  at  which  it  will  be  taken  up?  It  could  be  argued  that  there  is  a  simple  feedback  loop  here, 
as  shown  below.  As  the  number  of  information  centres  using  the  service  Increases,  the  number  of 
potential  recruits  goes  down  (since  only  a  limited  nuafcer  of  centres  exists  to  take  up  the  service). 


indicated  by  the  +  and  -  signs.  But  it  can  be  argued  alternatively  that  we  have  omitted  one  important 
consideration.  For  potential  users  actually  to  take  up  a  service,  they  must  obviously  know  that  it 
exists.  The  more  information  centres  make  use  of  a  service,  the  more  widely  it  is  known;  so  there  is  a 
positive  feedback  loop,  as  well  as  a  negative  one,  and  the  loop  now  becomes  slightly  more  complicated. 


This  simple  model  can  be  run  on  a  computer  to  see  how  various  assumptions  about  the  effectiveness  of 
publicity,  resulting  from  growth,  can  affect  the  growth  itself. 


System  dynamics  models  provide  a  convenient  basis  for  considering  some  of  the  fundamental  queries  that 
modelling  procedures  raise.  Firstly,  there  is  an  obvious  assumption  that  we  are  dealing  with  a  closed 
system.  In  our  example,  the  loops  involve  only  information  centres  and  the  new  online  service.  It  is 
not  difficult  to  think  of  a  great  range  of  other  factors  that  might  Impinge  on  the  activities 
represented  by  the  loops,  and  that  have  been  excluded.  Secondly,  even  within  this  closed  system,  a 
number  of  implicit  assumptions  have  been  made.  For  example,  no  allowance  has  been  made  for  centres 
which  try  out  the  service  as  an  experiment  and  then  drop  it.  Again,  any  online  service  is  likely  to 
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begin  with  a  blaze  of  publicity  unrelated  to  the  nunber  of  information  centres  then  making  use  of  It. 
Thirdly,  many  of  the  parameters  In  the  system  have  to  be  chosen  in  a  fairly  arbitrary  way.  For  example, 
it  is  not  necessarily  easy  to  determine  how  many  information  centres  are  available  to  absorb  the 
service,  or  to  what  extent  take-up  of  the  service  by  one  information  centre  will  influence  the  take-up 
by  another  centre.  There  are  two  fundamental  difficulties  here.  The  first  is  the  level  of  complexity 
to  which  the  system  should  (or,  indeed,  can)  be  pushed;  the  second  is  limitations  of  knowledge 
concerning  the  parameters  involved.  These  paints  afflict  almost  any  kind  of  modelling  process. 

APPLYING  MCOaS 

In  practice,  models  are  often  used  to  try  out  ideas  and  to  see  which  are  the  Important  variables,  rather 
than  to  derive  definitive  results.  From  this  viewpoint,  modelling  can  best  be  regarded  as  a  comparative 
process,  looking  at  what  differences  result  from  Introducing  different  sets  of  parameters  and  variables. 
Socio-economic  models  are  therefore  frequently  employed  as  guides  to  analysis  rather  than  full-blown 
tools  of  analysis  in  their  own  right.  This  approach  is  encouraged  by  the  fact  that  different  people 
modelling  the  same  system  may  regard  different  elements  of  it  as  being  most  significant.  For  our 
present  purposes,  it  is  probably  most  useful  to  look  at  the  types  of  model  that  might  be  employed  in 
assessing  an  information  centre  or  service,  and  then  to  mention  briefly  applications  of  some  of  the  more 
popular  app: caches. 

The  most  widely  used  model  is  based  on  some  kind  of  flow  diagram  (as  in  the  Shannon  and  Weaver  model). 
Such  models  typically  contain  nodes  linked  together  by  conaunicatlon  channels  (in  the  Shannon  and  Weaver 
model,  the  sender  and  receiver  of  a  message  are  the  nodes,  whilst  the  telephone  line  is  the  channel). 
Flow  diagrams  contain  four  basic  elements  -  the  nodes,  the  channels,  the  processes  involved  and  the 
messages.  Any  of  these  elements  may  receive  particular  emphasis  in  a  given  model,  depending  on  the 
needs  of  the  situation.  Both  the  Shannon  and  Weaver  model  and  the  system  dynamics  model  are  organised 
round  the  concept  of  information  flows.  As  this  suggests,  flow  models  can  be  either  qualitative  or 
quantitative  in  approach,  though  qualitative  usages  are  commoner. 

In  terms  of  classifying  flow  models,  the  presence  or  absence  of  feedback  is  probably  the  most  Important 
characteristic.  For  example,  the  Shannon  and  Weaver  model  has  been  applied  to  the  study  of  the  mass 
media.  In  this  case,  the  source  might  be  a  speaker  on  a  radio  programme  whose  voice  is  carried  via 
radio  waves  (the  channel)  to  listeners  in  the  home.  The  usual  flow  diagram  here  does  not  involve 
feedback.  A  linear  model  is  used  in  which  the  speaker  influences  the  listener,  but  not  vice  versa.  In 
a  telephone  conversation,  on  the  contrary,  each  participant  can  Influence  the  other;  so  any  model  of  the 
process  must  Incorporate  a  feedback  loop.  It  might  be  argued  that  the  direct  Information  transfer 
handled  by  a  centre  involved  feedback,  but  some  services  do  not.  However,  alnrast  any  information- 
handling  activity  requires  the  goodwill  of  its  users,  and  so  should  include  a  monitoring  element  that 
provides  some  level  of  feedback.  This  monitoring  is  Important  not  Just  for  gauging  user  satisfaction. 
When  radio  audiences  were  finally  investigated,  it  was  found  that  acceptance  of  broadcast  information 
did  not  follow  the  model  broadcasters,  themselves,  had  supposed.  It  transpired  that  some  listeners  not 
only  received  information  from  the  media,  but  actively  disseminated  it  to  friends  and  neighbours.  This 
secondary  dissemination  often  reached  a  wider  audience  than  the  original  broadcast. 

Many  information  centres  are  Involved  in  providing  a  service  not  only  for  users  of  information,  but  also 
for  producers  of  information  (who  form  part  of  their  user  community).  In  this  case,  the  feedback  loop 
can  be  so  strong  that  the  whole  conaunication  process  is  better  portrayed  as  a  single  continuous  loop. 
The  process  of  communication  can  be  divided  into  four  phases  -  the  generation  of  information,  its 
recording,  dissemination  and  utilisation.  But,  where  producers  of  information  are  present,  the  utilis¬ 
ation  of  Information  often  leads  to  the  generation  of  new  information.  So  information  flow  must  be 
modelled  as  a  cyclical  process.  For  exanqile,  the  production  and  use  of  reports  can  be  portrayed  in 
these  terras.  Typically,  a  series  of  stages  can  be  discerned,  starting  with  research  and  development, 
then  continuing  through  composition  and  recording,  printing  and  distribution,  acquisition  and  storage, 
organisation  and  control,  dissemination  and  presentation,  to  assimilation  which  then  leads  on  to  a  new 
stage  of  research  and  development.  This  kind  of  approach  concentrates  on  the  processes  operating.  An 
alternative  approach  concentrates  on  the  conmunlcatlon  channels  used  and  their  comparative  importance. 
For  example,  several  papers  have  examined  the  different  paths  that  information  can  follow  from 
originator  to  user,  and  how  this  may  involve  different  media. 

One  point  many  of  these  models  have  in  coimiion  is  that  they  treat  information  flow  as  a  sequential 
process.  The  information  passes  through  successive  stages  of  handling  in  much  the  same  way  that  a  car 
passes  along  a  production  line  as  it  is  built.  This  can  sometimes  be  a  fair  approximation  to  what 
happens,  but  Information  has  a  tendency  not  to  build  Itself  up  in  neat  packages.  Hence,  sequential 
models  of  information  flow  need  to  be  applied  with  care. 

Another  point  that  requires  consideration  is  whether  it  is  better  to  apply  a  static  or  a  dynamic  model. 
Although  the  use  of  the  term  'information  flow’  presupposes  motion,  many  studies  of  information  transfer 
assume  that  it  occurs  through  a  relatively  static  Infrastructure.  Information  diffuses  through  the 
unchanging  organisational  structure  in  much  the  same  way  that  water  trickles  through  a  sponge.  The 
alternative  viewpoint  is  to  suppose  that  the  passage  of  Information  is  a  dynamic  activity  with  the 
Infrastructure  adapting  itself  as  it  interacts  with  the  transfer  of  information.  The  difference  between 
the  two  viewpoints  is  especially  obvious  in  studies  of  communication  in  organisations.  One  approach  is 
to  take  the  staff  hierarchy  laid  down  by  the  organisation,  and  then  to  observe  how  information  moves 
vertically  and  horizontally  through  this  structure.  The  other  is  to  trace  the  flow  of  information 
through  an  organisation,  determine  >d!o  are  the  key  receivers  and  transmitters  of  information,  and 
examine  how  they  operate.  When  the  two  approaches  have  been  tried  in  tandem,  it  has  usually  been  found 
that  the  information  flows  only  partly  follow  the  formal  structure  of  the  organisation.  Most 
organisations  have  members  at  all  levels  in  the  hierarchy  who  act  as  major  nodes  for  information 
transfer  in  the  organisation.  The  existence  of  such  'gatekeepers'  Illustrates  the  need  to  be  flexible 
in  the  application  of  models.  The  choice  of  model  can.  Itself,  partly  determine  the  outcome  of  an 
investigation  of  performance. 

The  distinction  between  static  and  dynamic  models  can  be  related  to  the  need  to  consider  whether 
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infonutlon  transfer  Is  tiae-dependent.  A  snapshot  of  what  is  happening  at  a  particular  point  In  tine 
obviously  does  not  entail  a  discussion  of  trends.  But  it  is  usually  necessary  to  allow  for  any 
variation  with  tine  in  order  to  generalise  the  snapshot.  For  exanple,  coanunlcation  habits  vary 
throughout  the  day  -  direct  oral  coaaunicttlon  over  lunch,  telephone  calls  in  the  morning  and  afternoon, 
reading  of  reports  in  the  evening  -  so  sampling  of  coanunication  within  an  organisation  must  take  this 
into  account.  In  addition,  the  amouht  and  type  of  information  flow  can  depend  on  the  channel  used. 
Formal  and  informal  communication,  for  example,  often  have  to  be  modelled  in  different  ways. 

Conceptual  models  may  have  a  physical  analogue.  The  flow  model,  for  example,  is  obviously  analogous  to 
the  flow  of  water  through  a  pipe,  or  of  electricity  through  a  wire.  Another  such  analogue  that  lies  at 
the  basis  of  a  number  of  models  is  diffusion.  When  a  solid  dissolves  in  a  liquid,  the  amount  present  at 
each  point  in  the  liquid  gradually  increases  with  time.  By  analogy,  information  can  be  thought  of  as 
appearing  at  one  point  within  a  system  and  then  diffusing  gradually  throughout  the  system. 
Alternatively,  this  process  of  Information  transfer  can  be  compared  with  the  progress  of  an  epidemic, 
with  each  new  'infection'  corresponding  to  someone  who  has  just  heard  the  item  of  information.  These 
analogies  can  be  expressed  in  mathematical  form  and  used  to  provide  quantitative  results;  but  they  are 
more  often  used  as  qualitative  guides  to  what  is  happening. 

MATHEMATICAL  ANALYSIS 

Most  applications  of  mathematics  to  information  transfer  problems  Involve  statistics  or  probability.  A 
good  example  is  provided  by  the  extensive  studies  of  inter-library  loans  which  subject  the  collected 
data  to  various  kinds  of  statistical  handling,  including  trend  analysis.  The  techniques  are  often 
standard  in  socio-economic  modelling:  the  only  point  requiring  care  in  information  work  is  that  heavy¬ 
weight  computer  methods  should  not  be  applied  to  data  that  do  not  warrant  them.  Such  quantitative 
modelling  works  very  well  when  cost  comparisons  are  involved.  A  typical  case  study  might  be  a  cost 
comparison  of  online  and  manual  bibliographical  information  retrieval.  Take,  for  example,  a  simple 
exercise  of  this  type  carried  out  by  Flynn  et  al  (2).  This  works  by  taking  time  spent  on  the  activities 
and  converting  this  to  money  terms.  The  total  time  for  an  online  search  is  the  sum  of  the  average  time 
spent  per  search  in  search  strategy  formulation  plus  the  time  spent  per  search  at  the  terminal  plus  the 
time  spent  per  search  in  post  search  analysis  and  editing.  This  can  be  converted  into  money  terms  via 
the  salary  of  the  searcher.  To  the  results  must  be  added  the  various  search  charges  involved.  The 
final  figure  can  then  be  compared  with  the  equivalent  manual  costs.  In  the  specific  example  given,  the 
total  online  cast  was  t2i.S0,  compared  with  a  total  manual  cost  of  t26.70.  This  gave  for  the  cost 
effectiveness  of  online  searching  compared  with  manual  searching  a  figure  of  24  per  cent.  At  first 
sight,  this  is  a  straightforward  calculation  involving  no  modelling.  However,  the  result  actually 
depends  on  how  the  search  is  carried  out  and  by  whom.  A  search  carried  out  by  a  senior  research 
scientist  has  to  be  costed  out  in  a  different  way  from  one  performed  by  an  information  scientist. 
Indeed,  Flynn  et  al  end  their  paper  by  describing  a  general-purpose  cost  model. 

A  related  area  is  trend  analysis.  This  also  employs  well-tried  quantitative  methods,  but  suffers  from 
the  usual  problem  in  information  work  that  its  scope  for  application  is  often  limited.  It  can  best  be 
applied  to  the  study  of  specific  activities.  More  generally,  extrapolation  of  trends  can  help  with 
certain  types  of  problem.  For  example,  suppose  calls  on  an  Information  service  are  increasing.  To 
remain  effective  in  terms  of  response  to  user  demand,  it  may  be  necessary  to  recruit  additional  staff. 
But  because  staff  come  in  discrete  units,  additional  staff  may  be  underworked  until  demand  has  grown 
still  further.  Hence,  the  change  may  enhance  the  effectiveness  of  the  service,  whilst  temporarily 
depressing  Its  degree  of  economy  and  efficiency.  A  proper  study  of  trends  can  ensure  that  this  effect  Is 
reduced  to  a  minimum.  But,  unless  care  is  taken,  this  may  be  putting  the  cart  before  the  horse. 
Extrapolation  forward  basically  Implies  a  'more  of  the  same'  approach.  At  some  stage,  a  manager  must 
decide  where  the  service  ought  to  be  going.  Analysis  of  trends  can  indicate  whether  it  is  likely  to  get 
there,  but  not  what  that  goal  should  be.  Hence,  the  start  point  is  a  qualitative  mission  statement, 
which  can  then  be  translated  into  quantitative  terms. 

FORECASTING 

Attempts  to  forecast  the  future  typically  include  both  qualitative  and  quantitative  data.  The  balance 
between  the  two  varies.  A  service  that  has  been  in  operation  for  some  years  may  have  accumulated  a 
considerable  amount  of  quantitative  data.  But  if  it  is  a  question  of  starting  up  a  new  service  from 
scratch,  there  may  be  little  in  the  way  of  quantitative  data  to  use.  As  the  contingent  factors  that  need 
to  be  considered  increase,  so  quantitative  methods  become  more  suspect.  For  forecasts  aimed  at  the 
immediate  future  (say,  2-3  years),  trend  extrapolation  is  usually  safe  enough.  Longer-term  forecasts 
(say,  5-10  years)  run  up  against  the  difficulty  that  the  service  can  no  longer  be  considered  as  an 
Isolated  entity.  The  environment  in  which  the  service  operates  changes  over  this  period  of  time, 
usually  in  ways  too  complex  for  quantitative  extrapolation. 

There  is  a  considerable  literature  on  longer-range  forecasting,  but  much  of  it  relates  to  technology. 
Indeed,  there  is  often  an  underlying  assuaption,  explicit  or  implicit,  that  new  technology  will  provide 
much  of  the  drive  for  future  change  in  the  information  world.  For  example.  Lesser  and  Yagionos  (3)  have 
put  forward  the  two  models  shown  below  for  the  provision  of  information  services.  The  first  illustrates 
the  traditional  flow  of  information;  the  second  indicates  how  this  is  expected  to  change  under  the 
influence  of  new  technology.  The  changes  are  based  on  qualitative  extrapolation  of  present  trends  -  for 
example,  that  information  will  became  more  easily  copied,  more  easily  deliverable  and  more  easily 
manipulated  than  at  present.  On  the  basis  of  this  qualitative  assessment,  important  changes  are  forecast 
for  the  future:  in  particular,  that  information  services  will  become  more  central  to  the  flow  of 
information  than  hitherto  and  that  the  links  in  the  communication  chain  will  alter. 

Technology-driven  extrapolations  tend  to  assume  that,  because  technology  offers  a  particular  facility, 
that  facility  will  be  used.  Though  this  is  often  true,  there  are  plenty  of  counter-examples  to  show 
that  users  will  not  necessarily  take  what  appears  to  be  the  logical  path.  As  with  most  forward  looks, 
the  extrapolation  needs  to  be  combined  with  some  assessment  of  where  the  total  environment  is  going. 
One  popular  method  of  doing  this  is  via  a  scenario-writing  approach. 


'The  term  "scenario-writing”  denotes  a  technique  which  attenpts  to  set  up  a  logical  sequence 
of  events  in  order  to  show  how,  starting  froa  the  present  (or  any  other  given)  situation,  a 
future  state  might  evolve  step  by  step  ...  scenario-writing  may  be  regarded  as  an  extension 
of  contextual  mapping  to  the  formation  of  a  synoptic  view  of  as  many  developments  as  can  be 
grasped  and  as  may  appear  relevant  to  an  experimental  simulation  of  a  possible  reality'  (4). 

The  aim  of  this  approach  is  therefore  not  to  produce  a  single  picture  of  where  things  are  going,  but 
rather  to  map  out  a  series  of  possible  pictures  of  the  future  and  the  paths  to  them.  For  example,  some 
time  ago  Asllb  looked  at  the  possibility  of  using  a  scenario  approach  to  examine  the  Impact  on  new 
technology  of  library  and  Information  manpower  in  the  U.k.  (5).  Three  scenario  were  chosen.  The  first 
supposed  that  future  changes  would  follow  current  trends,  so  the  future  impact  of  technology  on  manpower 
could  be  extrapolated  directly  from  what  is  happening  now.  The  second  postulated  that  a  change  in 
services  would  take  place.  Developments  would  go  ahead  as  in  the  first  scenario,  but  resultant  Job 
losses  would  be  offset  by  the  establishment  of  new  types  of  services.  The  third  scenario  postulated  the 
opposite  effect.  New  automated  services  would  appear,  but  they  would  act  to  displace  manpower  at  an 
even  more  rapid  rate. 

This  example  demonstrates  the  basic  properties  of  scenario-writing.  They  are  designed  to  bracket  the 
likely  future  possibilities,  and  they  allow  room  for  a  range  of  factors  to  be  included  along  with 
technology.  In  consequence,  examining  a  scenario  gives  a  feel  for  the  whole  range  of  factors  involved. 
Perhaps  more  Importantly,  it  allows  a  comparison  between  possible  trends  and  desired  objectives,  and  so 
indicates  the  factors  which  may  hinder,  or  facilitate,  reaching  these  objectives.  A  full  scenario- 
writing  study  can  be  very  time-consuming,  but,  as  with  most  approaches  to  modelling,  elements  of  the 
method  can  be  used  for  a  more  rapid  and  superficial  evaluation.  The  approach  does  emphasize  the 
Importance  of  considering  both  qualitative  and  quantitative  factors  in  making  an  assessment.  This  point 
can  be  seen  even  more  clearly  in  the  application  of  a  systems  approach  to  modelling  Information  handling 
in  an  organisation. 

THE  SYSTEMS  APPROACH 

A  'system'  is  a  group  of  related  elements  organised  for  a  particular  purpose.  Systems  can  be  physical, 
biological,  social,  etc.,  but  the  ones  of  most  interest  to  information  work  are  a  combination  of  the 
social  and  technological.  Many  of  the  operations  carried  out  by  information  centres  or  services  can  be 
described  as  'systems'.  One  advantage  in  doing  so  Is  that  such  an  approach  can  lead  naturally  to  a 
definition  of  performance  measures  for  the  system  concerned.  A  distinction  must  be  drawn  between  two 
methods  of  tackling  systems  -  the  'hard'  and  the  'soft'.  The  hard  systems  methodology  is  based  on  an 
approach  cornnon  in  engineering,  which  is  especially  aimed  at  the  management  of  change.  It  reflects  the 
need  to  structure  the  sort  of  messy  problems  encountered  in  real  life,  so  that  they  can  be  tackled  in  a 
logical  sequence  of  steps.  These  can  be  listed  as  follows: 

(t)  build  up  a  comprehensive  picture  of  the  situation  and  the  Issues  involved; 

(2)  set  objectives  for  the  change  which  accurately  reflect  the  concerns; 

(3)  design  a  range  of  options  or  alternative  strategies  which  meet  the  objectives. 

(4)  manage  the  choice  between  alternative  options; 

(5)  development  a  strategy  for  implementing  the  changes. 

The  model  based  on  this  list  is  shown  below:  note  that  it  represents  not  the  system  itself,  but  a  way  in 
which  the  system  can  be  examined. 


SEqUENCE  OF  STEPS  IN  THE  HARO  SYSTEMS  APPROACH 

The  diagram  seems  to  show  a  linear  progression  in  tackling  the  problem.  In  practice,  things  are  less 
simple  because  working  on  the  problem  usually  brings  new  insights,  and  these  have  to  be  fed  back  to 
earlier  steps  in  the  model.  So  application  of  the  model  is  generally  iterative.  In  fact,  the  hard 
systems  methodology  is  Intended  for  use  especially  by  consultants,  who  are  frequently  faced  by  clients 
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who  alter  their  minds  and  so  entail  alterations  to  the  parameters  as  the  work  progresses. 

The  stages  In  the  process  can  be  stamurlsed  as  follows: 

(1)  System  description.  This  sets  out  what  the  project  Is  about  and  what  the  problem  that  has  been 
Identified  Is.  This  leads  to  problem  perception,  which  lays  out  who  Is  Involved  In  the  project,  to 
what  extent,  and  what  their  perceptions  of  the  problems  are.  The  next  step  Is  to  decide  what  are 
the  basic  elements  Involved  In  the  problem.  Some  will  be  descrlbable  as  systems,  but  some  will  be 
non-systemlc.  The  latter  must  be  separated  out,  and  a  decision  made  on  which  of  the  former  should 
be  studied  In  detail. 


(2)  Identification  of  objectives  and  constraints.  The  former  say  where  you  want  to  get  to,  and  the 
latter  what  holds  you  back.  The  objectives  often  Include  Items  over  which  you  have  some  control, 
whereas  the  constraints  are  Imposed  by  the  environment  and  are  often  beyond  control.  The 

objectives  must  be  chosen  In  such  a  way  as  to  provide  an  Indication  of  what  action  Is  needed  for 
them  to  be  attained. 


(3)  Formulate  measures  of  performance.  The  hard  systems  approach  stresses  the  need  to  devise  ways  of 
measuring  whether  objectives  are  being  reached.  What  measure  Is  appropriate  for  a  particular  area 
of  performance,  and  what  target  level  of  performance  should  be  aimed  at  after  what  period  of  time? 
For  example,  suppose  the  objective  is  to  Increase  the  number  of  users  of  a  commercial  online 
service.  The  objective  might  be  formulated  as  a  requirement  for  a  20  per  cent  increase  in  the 
number  of  subscribers  over  the  next  twelve  months.  There  are  frequently  several  different 

potential  ways  of  attaining  a  given  objective,  and  the  need  Is  to  compare  them.  For  this,  it 
obviously  helps  to  have  all  the  performance  measures  expressed  in  the  same  terms.  The  most  popular 
common  quantity  is  money,  but  it  cannot  be  applied  universally.  Speed  of  delivery  of  a  service, 
for  example.  Is  obviously  most  sensibly  expressed  In  time  terms. 


(4)  Generation  of  routes.  Objectives  can  usually  be  arranged  in  a  hierarchy,  with  the  lower-order 
objectives  contributing  to  the  fulfilment  of  the  higher-order  objectives.  The  important  action  at 
this  stage  is  to  expand  the  hierarchy,  so  that  the  higher-order  objectives  can  be  reached  by  as 
many  different  routes  as  possible.  Then  the  list  can  be  narrowed  down  to  produce  a  set  of  most 
acceptable  routes,  given  the  existing  constraints. 


(5)  Model  construction.  The  formulation  of  a  list  of  routes  Is  equivalent  to  setting  up  a  model  of  the 
system.  According  to  the  hard  systems  approach,  each  route  should  be  labelled  with  a  statement  of 
how  performance  will  be  measured  at  each  point.  If  the  system  being  studied  is  fairly  simple,  the 
relationships  and  numbers  in  the  final  model  can  be  worked  out  by  hand.  For  more  coiqilex  systems, 
computer  modelling  may  be  needed.  It  Is  typically  at  this  stage  that  the  assumptions  built  into 
the  model  are  tested.  If  they  are  found  wanting,  it  may  be  necessary  to  go  back  to  the  beginning. 


(6)  Evaluation.  This  Is  partly  an  extension  of  the  previous  stage.  It  Involves  a  full  testing  of  the 
model  to  explore  Its  sensitivity  to  variations  both  In  the  assuaptlons  it  Is  based  on  and  to  the 
parameters  which  are  Input  to  It.  But  it  then  leads  on  to  a  quantitative  comparison  of  the 
different  routes  to  the  objectives  together  with  an  assessment  of  what  are  the  virtues  and  defects 
of  each  route. 


(7)  Choice  of  routes.  The  clients  and  the  consultants  get  together  at  this  stage  to  decide  which 
options  should  be  selected  for  implementation.  In  many  cases,  there  is  no  single  'best'  route: 
each  will  have  advantages  and  disadvantages.  At  this  point,  non-quantitative  factors  (perhaps  of 
the  sort  that  were  discarded  at  the  beginning  as  non-systemlc)  may  decide  which  route  is  to  be 
taken. 


(8)  Implementation.  The  decision  on  which  option  to  take  must  be  based  In  part  on  a  feeling  about  what 
it  is  feasible  to  Implement  in  practice.  For  exaaple,  some  routes  might  seem  advantageous  to 
management,  but  be  unacceptable  to  the  workforce.  This  requires  decisions  as  to  the  strategy  to  be 
employed  in  introducing  the  recanmended  changes.  Which  part  of  the  plan  shall  be  Introduced  when, 
and  what  will  the  overall  implications  of  the  changes  be? 


This  description  of  a  hard  systems  approach  may  be  contrasted  with  an  equivalent  soft  systems  approach. 
The  latter  is  concerned  with  description  and  qualitative  factors,  rather  than  with  quantitative 
measures.  It  is  similar  to  a  hard  systems  approach,  however,  in  emphasising  a  step-by-step  structured 
analysis  allowing  plenty  of  scope  for  iteration.  The  model  used  'n  this  kind  of  analysis  is  shown  in 
the  diagram  below,  and  can  be  described  along  the  lines  of  the  hard  systems  approach. 
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STAGES  OF  THE  SOFT  SYSTEMS  KTHODOLOGY.  REAL  MORLD  (top) ,  ABSTRACT  UORLO  (bottoa) 

(1)  Ttie  problem  situation.  This  forms  the  initial  stage  of  finding  out  as  much  as  possible  about  the 
problem  to  be  studied,  especially  by  talking  to  the  people  concerned. 

(2)  The  situation  analysed.  The  information  so  gathered  is  brought  together  in  order  to  formulate  a 
picture  of  what  is  going  on.  The  word  'picture'  is  to  be  taken  literally  here,  for  a  key  element 
Involves  drawing  a  'rich  picture'  -  a  kind  of  cartoon  which  brings  together  all  the  elements,  hard 
or  soft,  which  appear  to  be  relevant  to  the  problem.  The  elements  may  either  relate  to  structure 
(factors  that  change  only  slowly  with  time,  such  as  the  institutional  hierarchy  or  the  physical 
layout  of  the  building  which  houses  the  organisation)  or  to  process  (factors  that  change  rapidly 
with  time,  as  many  organisational  activities  do).  The  rich  picture  is  examined  to  determine  two 
things.  The  first  is  the  primary  tasks  -  the  tasks  which  the  organisation  was  created  to  perform, 
and  those  which  it  must  perform  in  order  to  keep  Itself  running.  The  second  Involves  issues.  These 
are  the  Items  of  concern  which  form  the  basis  of  the  problem  to  be  solved. 

(3)  Relevant  systems.  At  this  stage,  the  investigation  moves  from  the  concrete  to  the  abstract.  Each 
problem  that  has  been  isolated  in  the  rich  picture  is  examined  to  see  whether  it  can  be  concep¬ 
tualised  as  some  form  of  system.  If  so,  it  is  classified  as  a  'relevant'  system.  Several  such 
relevant  systems  can  typically  be  devised  for  each  problem  theme.  The  essential  characteristics  of 
each  relevant  system  are  then  encapsulated  in  a  'root  definition'  (a  description  in  a  couple  of 
sentences). 

(4)  Conceptual  model.  This  Involves  deciding  what  is  the  minimum  set  cf  activities  that  will  fulfill 
the  requirements  contained  in  the  root  definitions.  The  list  of  activities  is  then  put  in  order, 
so  that  each  follows  In  logical  order.  This  exercise  usually  leads  to  the  drawing  of  a  diagram 
(representing  the  model)  which  traces  the  logical  consequences  of  following  a  particular  relevant 
system. 

(5)  Comparison.  At  this  point,  the  abstract  model  is  ccmpared  with  the  original  problem  situation. 
There  Is  usually  a  mismatch  between  the  type  and  sequence  of  activities  visualised  by  the  model  and 
those  that  actually  occur.  Any  such  occurrences  are  itemised  at  this  stage. 

(6)  Debate.  How  important  are  the  mismatches  between  the  model  and  the  problem?  To  what  extent  can 
any  changes  suggested  by  the  model  be  implemented?  One  important  question  is  often  whether  the 
difficulties  are  general,  so  that  it  would  not  be  feasible  to  Implement  the  proposed  changes  under 
any  clrcunstances,  or  whether  they  are  peculiar  to  the  specific  situation. 

(7)  Implementation.  As  with  the  hard  systems  approach,  the  consultant  can  advise  the  clients,  but  is 
rarely  responsible  for  the  actual  process  of  implementation. 
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PERFORNMCE  «SURES  AH)  INDICATORS 

Tlie  hard  and  soft  systeas  have  been  described  In  scan  detail  because  they  highlight  a  nuad>er  of  the 
central  points  In  examining  the  functioning  of  an  information  centre  of  service.  Firstly,  both 
approaches  eagjhaslse  the  central  role  of  the  laodel  in  the  evaluation.  Secondly,  they  indicate  the 
limits  on  what  can,  and  what  cannot  be  measured.  Performance  measures,  as  usually  defined,  are  quanti¬ 
tative.  A  distinction  is  soantlmes  drawn  between  measures  and  Indicators.  The  former  represent  a 
direct  assesssKnt  of  an  activity.  The  latter  measure  (usually,  though  not  always,  quantitatively) 
some  parameter  which  Is  believed  to  be  related  to  the  activity.  For  example.  It  is  difficult  to  measure 
staff  morale,  or  even  define  it  in  a  measurable  way.  So  it  might  be  postulated  that  absenteeism,  which 
can  be  easily  defined  and  measured,  is  an  indicator  of  staff  morale. 

Three  types  of  activity  can  be  distinguished  where  measures,  or  indicators,  can  be  derived. 

(1)  Input  measures  -  such  as  stock  acquisition,  staff  time,  overheads. 

(2)  Intermediate  output  measures  -  such  as  enquiries  answered,  searches  done,  nuaber  of  users. 

(3)  Final  output  measures  -  measures  of  quality,  such  as  the  relevance  of  the  information  provided,  and 
measures  of  value,  such  as  the  amount  of  money  saved  by  a  user  as  a  consequence  of  obtaining 
information  from  a  service. 

The  coiR)arlson  of  input  with  Intermediate  output  measures  gives  an  indication  of  the  efficiency  of  the 
system,  since  efficiency  concerns  the  conversion  of  inputs  to  outputs  with  mlnimimi  wastage.  However, 
this  comparison  does  not  always  reflect  on  the  economy  of  the  system,  that  is  whether  the  information  Is 
being  provided  at  the  lowest  passible  cost.  A  comparison  with  final  output  measures  gives  the  effec¬ 
tiveness  of  the  system,  since  this  concerns  what  clients  gain  from  a  particular  level  of  service.  It  is 
worth  emphasising  that  there  are  many  variant  terms  and  usages  of  terms  in  this  field.  Thus  the 
distinction  drawn  previously  between  'measures'  and  'indicators'  Is  not  universally  accepted.  Some 
writers  use  'efficiency'  to  cover  both  efficiency  and  economy  as  defined  above.  Again,  the  types  of 
measure  can  be  divided  in  other  ways,  e.g.  Into  Input^rocesses/output.  Hence,  reading  papers  on  the 
measurement  of  performance  requires  some  attention  to  the  details  of  definition. 

A  further  point  to  be  noted  is  that  information  handling  is  a  trickier  Item  to  perform  measurements  on 
than  most  other  productive  activities.  From  this  viewpoint,  some  coements  by  Rice  and  Blair  (6)  are 
worth  quoting: 

'Information  does  not  itself  behave  like  physical  inputs  or  outputs.  Constrained  only  by  the 
physical  markers  used  to  transport  its  symbolic  content,  information  can  be  diffused  over 
time  and  space  in  ways  that  material  products  can  never  be.  IXi  the  one  hand,  its  utility  may 
expand  the  more  it  Is  used  by  more  people;  on  the  other,  it  may  lose  all  worth  when  other 
people  obtain  it.  The  quality  and  timeliness  of  Information  may  be  critical  to  Its  use;  yet 
quality  may  be  largely  subjective  and  determined  separately  by  the  producer  and  the  user. 
Overabundance  may  sap  the  scarce  resource  of  managerial  attention.  More  efficient  dissemin¬ 
ation  of  Information  that  is  incorrect  or  unneeded  will  clearly  harm  the  effectiveness  of  an 
organisation.  That  is.  the  effectiveness  or  outcome  of  many  information  activities  is  more 
crucial  than  Its  efficiency.' 

THE  OEVaOPfCHT  OF  PERFORMANCE  MEASURES 

In  view  of  the  particular  coieplexlty  of  performance  measurement  for  information  work,  it  is,  perhaps, 
less  surprising  that  much  more  has  been  written,  or  tried  out  on  an  experimental  basis,  than  has  bwn 
implemented  routinely  in  this  area.  The  current  emphasis  on  the  topic  can  probably  be  traced  back 
twenty  years  to  a  pioneering  book  by  Morse  (7).  This  dealt,  from  a  fairly  theoretical  viewpoint,  with 
library  effectiveness.  A  few  years  later,  an  influential  American  symposium  looked  at  approaches  to 
measuring  library  effectiveness  (8).  One  point  that  emerged  was  the  importance  of  examining  essentially 
qualitative  factors  (such  as  user  satisfaction),  rather  than  concentrating  on  quantitative  measures 
only.  There  was  considerable  discussion  of  what  quantitative  measurements  could  be  devised  which  would 
permit  Inferences  about  qualitative  factors  to  be  made  -  the  question  of  performance  Indicators,  as  we 
have  defined  them.  A  distinction  was  drawn  between  static  and  dynamic  measures.  It  was  felt  that 
librarians  were  then  more  concerned  with  static  measures  relating  to  stock,  rather  than  dynamic  measures 
relating  to  service.  By  1972,  over  500  articles  and  books  on  performance  measures  for  libraries  had 
already  been  published,  but  most  concentrated  on  a  few  basic  types  of  measure  -  accessibility,  cost, 
user  satisfaction,  response  time,  cost/benefit  and  use.  Where  applications  of  these  had  been  made,  they 
seemed  to  be  more  often  to  parts  of  the  system,  rather  than  to  the  library  as  a  whole.  Much  of  the 
emphasis  in  this  early  work  was  on  the  need  to  measure  effectiveness.  For  example,  Orr  looked 
particularly  at  how  the  goodness  of  library  services  might  be  measured  (9).  He  decided  that  the 
ultimate  criteria  for  evaluating  a  library  service  are  quality  (i.e.  how  good  the  service  is)  and  value 
(l.e.  how  much  good  it  does).  Another  characteristic  wB3  the  continuing  emphasis  on  mathematical 
sophistication.  Rodwell  blamed  that,  in  part,  for  the  lack  of  use  rf  performance  measures,  along  with 
the  cost  and  time  Involved  and  the  lack  of  motivation  (10).  White  backed  these  comaents,  noting  that 
the  emphasis  on  theory  of  performance  measures  tended  to  leave  out  the  human  element,  which  is  central 
to  much  library  and  Information  work  (11).  He  also  remarked  that  performance  measures  were  still  far 
more  talked  about  than  applied. 

Things  began  to  change  towards  the  end  of  the  1970s.  (For  an  up-to-date  survey  of  these  changes,  see 
Goodall  (12).)  In  the  U.K.,  local  government  had  become  Increasingly  interested  in  performance  measures 
during  the  decade.  In  1979,  putting  performance  review  into  practice  was  published.  It  set  out  guide¬ 
lines  for  the  measurement  of  local  government  activities,  ana  included  a  section  on  review  procedures 

and  output  assessment  for  libraries.  The  methodology  is  based  on  looking  at  the  needs  of  specific 
client  groups  and  measuring  the  extent  to  which  they  are  being  met.  In  1980,  the  American  Library 

Association  held  a  conference  on  Library  effectiveness:  a  state  of  the  art.  This  contains  a  paper  by 

Williams  which  reports  on  an  American  survey  or  public  libraries  employing  22  different  measures  to 
indicate  trends  and  relationships.  The  paper  underlines  a  point  frequently  made  elsewhere  that  a 
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considerable  fraction  of  the  data  traditionally  collected  by  library  and  information  services  are  of 
little  value  from  the  viewpoint  of  compiling  performance  measures.  A  clear  trend  during  this  period  was 
to  explore  convarative  measures  of  performance.  In  the  U.K. .  a  feasibility  study  in  1977,  followed  by  a 
field  trial  In  1979,  led  to  the  publication  of  Pilot  comparisons  for  public  libraries  (13). 

More  recently,  a  similar  comparison  has  been  made  of  academic  libraries  (14).  Such  comparisons  were 
expected  to  allow  Identification  of  areas  of  strength  and  weakness,  so  encouraging  attempts  to  improve 
performance.  It  had  already  been  found  for  a  variety  of  industrial  activities  that  comparative 
evaluation  provided  a  good  management  tool,  especially  if  carried  out  on  an  annual  basis.  The  procedure 
has  the  virtue  that  like  is  being  compared  to  like,  and  the  defect  that,  if  the  standard  is  low  in  all 
the  institutions  surveyed,  a  comparative  evaluation  will  not  reveal  the  fact.  Much  of  the  relevant  work 
on  performance  measureaKnt  has  been  aimed  at  standardisation,  so  that  the  same  procedures  can  be  used  by 
a  range  of  organisations.  In  1985,  the  U.S.  Office  of  Management  Studies  looked  at  performance  measures 
for  academic  and  research  libraries.  The  report  notes  some  familiar  points;  for  example,  that  the  more 
sophisticated  methods  for  deriving  performance  measures  are  generally  too  complicated  for  ordinary  use. 
and  that  the  definition  of  objectives  by  libraries  in  such  a  way  that  their  achievement  can  be  readily 
measured  is  rare.  The  Office  describes  a  model  for  measuring  performance  which  takes  into  account  the 
kinds  of  measurement  made,  user  expectations,  standards,  objectives  and  evaluation  of  services.  The 
problem  of  lack  of  clearly  defined  objectives  was  noted  also  for  public  libraries  in  a  British  report  on 
A  costing  system  for  public  libraries  (15).  This  recent  report  found,  however,  that  all  types  of 
library  are  now  taking  the  application  of  performance  measures  more  seriously.  This  was  perceived  to  be 
in  response  to  the  Increasing  pressures  due  to  (1)  their  perception  of  declining  resources;  (2)  a 
perceived  demand  for  an  Improved  level  of  service;  (3)  an  increasing  demand  for  accountability. 
Librarians'  interest  was  found  to  centre  on  deriving  finance-related  data,  especially  concerning  cost 
centres,  functional  costing,  client  group  costing,  performance  Indicators  and  income.  The  report  also 
notes  that  several  simple  methods  of  measuring  performance  are  now  available  for  application.  In 
addition,  the  growing  use  of  conputer  systems  in  libraries  is  making  it  increasingly  easy  to  collect 
certain  types  of  data  which  can  be  employed  in  constructing  performance  measures. 

This  last  point  is  Important  because  it  is  closely  tied  to  the  question  -  why  are  performance  measures 
for  library  and  information  work  so  little  applied?  It  Is  worth  quoting  a  relatively  recent  publication 
on  cost  management  for  libraries  (16). 

■The  practice  of  costing  library  and  information  operations  and  the  principles  and  techniques 
available  to  support  the  economic  management  of  library  and  information  services,  are  one  of 
the  weakest  areas  In  the  repertoire  of  library  management  ...  In  addition,  the  actual 
machinery  (consisting  of  procedures,  conventions,  systems,  forms  and  specifications)  will  be 
found  to  be  relatively  underdeveloped  compared  to  the  conventional  business  enterprise, 
whether  It  Is  Industrial  or  comaercial.  public  or  private  sector.' 

The  reasons  why  performance  measures  and  Indicators  are  not  used  in  library  and  information  work  prove 
on  investigation  to  be  extensive.  They  may  be  suimarised  as  follows; 

(1)  Librarians  may  not  be  sufficiently  motivated  to  use  performance  measures.  This  may  be  because  they 
have  survived  happily  without  them  so  far,  or  because  they  cannot  afford  the  extra  effort  required 
in  their  present  straightened  circumstances,  or  because  their  employers  are  opposed  to  the  use  of 
such  measures. 

(2)  The  effort  may  not  appear  to  be  worthwhile  because  changes  suggested  by  the  performance  measures 
cannot  be  executed  owing  to  limited  resources. 

(3)  The  measures  may  be  seen  as  inadequate  or  inapplicable,  and  it  may  not  be  clear  how  they  can  be 
used  to  establish  standards.  Alternatively,  libraries  may  not  know  that  suitable  measures  are 
available. 

(4)  It  may  seem  that  simple  measures  only  give  a  misleading  picture,  whilst  coeplex  measures  may 
require  too  much  expenditure  of  time  and  money  for  regular  use. 

(5)  The  results  from  evaluation  of  performance  measures  can  work  either  way.  In  some  cases,  they  may 
underline  the  efficiency  of  a  service;  in  others,  they  may  suggest  the  opposite.  Librarians 
may  be  more  worried  by  the  latter  possibility  than  heartened  by  the  former. 

mSUMK  PERFOWMCE 

As  the  survey  of  past  work  illustrates,  there  has  always  been  a  belief  that  effectiveness  is  a  more 
important  measure  in  information  work  than  efficiency.  The  difficulty  is  that  factors  relating  to 
effectiveness  are  difficult  to  define  in  quantitative  terms  and  it  often  requires  expenditure  of 
considerable  time  and  energy  to  collect  the  necessary  data.  As  an  example  of  the  type  of  problems 
involved,  we  may  consider  a  study  recently  completed  at  Loughborough  (17).  This  examined  the  question 
of  measuring  the  Impact  of  new  information  technology  on  the  provision  of  information  to  users.  It  is 
not  difficult  to  formulate  a  series  of  questions  that  need  to  be  a"swered  in  this  context.  For  example, 
one  list  might  be; 

(1)  How  does  the  Introduction  of  new  coariunlcatlon  technologies  into  an  organisation  change  the 
existing  patterns  of  organisational  conmunlcatlon? 

(2)  To  what  extent  do  the  new  office  technologies  support  or  subvert  the  organisational  structure  as  it 
channels  coaiminlcation  flows? 

(3)  How  will  the  new  technologies  affect  office  productivity? 

(4)  Do  the  new  office  technologies  have  greater  effects  (1)  on  certain  individuals  in  an  organisation 
than  on  others;  (il)  on  certain  organisations? 
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(5)  How  does  the  networking  nature  of  the  new  office  technologies  affect  their  acceptance  and  use? 

(6)  How  can  the  new  office  technologies  be  redesigned  so  as  to  becone  more  acceptable  and  nore  useful? 

(7)  How  iwportant  is  the  name  of  a  new  cosBunlcatlon  technology  as  a  factor  in  its  acceptance? 


An  examination  of  this  list  suggests  how  many  gualltatlve  and,  indeed,  subjective  factors  are  involved 
in  assessing  change.  Nevertheless,  most  can  be  converted  to  a  form  which  permits  a  more  objective 
comparison.  For  example,  a  study  of  office  automation  in  Sermany  Isolated  by  qualitative  methods  four 
factors  which  seemed  especially  to  affect  willingness  to  use  new  communication  channels  (18).  These 
were  labelled  managerial  promptness,  semantic  complexity,  interpersonal  confidence  and  adainistrative 
accuracy.  An  opinion  poll  of  likely  users  of  the  new  channels  led  to  a  weighted  assessment  of  how  each 
channel  was  seen  in  terms  of  these  four  factors.  The  resulting  diagram  makes  it  easy  to  compare  the 
properties  of  new  and  traditional  coaisunication  channels,  whilst  emphasising  whichever  of  the  four 
factors  seems  most  important  in  a  given  situation. 
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What  such  studies  of  the  impact  of  new  technology  underline  is  the  need  to  be  clear  about  organisational 
objectives  and  how  they  are  achieved.  In  particular,  it  is  necessary  to  be  clear  what  the  performance 
measures  and  indicators  are  to  be  used  for  before  trying  to  select  them.  Suppose  -  as  is  often  the  case 
-  that  they  are  to  be  used  to  look  for  the  benefits  of  introducing  changes  into  the  system.  Then,  it  is 
obviously  necessary  to  have  a  full  list  of  possible  benefits  at  the  start.  The  Central  Computer  and 
Telecommunications  Agency  of  H.H.  Treasury  has  tried  to  draw  up  a  list  (reproduced  below).  This  con¬ 
tains  40  factors  placed  in  related  groups  of  five.  The  table  reflects  very  well  t(ie  spread  of  measure¬ 
ments  that  would  be  necessary  in  order  to  do  a  thorough  Job  of  assessing  benefits.  The  Agency  divides 
the  factors  into  three  types  according  to  their  ease  of  measurement.  These  can  be  paraphrased  as  being 
(1)  quantitative,  (2)  qualitative,  but  related  quantitative  Indicators  might  be  found,  and  (3) 
qualitative.  Hardly  surprisingly,  most  benefit  assessment  so  far  has  concentrated  on  factors  in  the 
first  column. 


POTENTIAL  BENEFITS 

MOST  LIKELY 

INVESTMENT  APPRAISAL 

CATEGORY 

Capable  of  quantifi- 

Generally  quantifi- 

Identifiable, 

cation  and  valuation 

flable,  difficult  to 

but  not 

value 

quantifiable 

Work  downgraded 
Overtime  reduced 
Travelling  time  and  cost  reduced 
Consultancy/agency  use  reduced 


Use  of  bureau  services  reduced 
Consumables  reduced 
Less  accommodation  required 
Nanning  for  peaks  avoided 
Sreater  accuracy 


X 

X 

X 

X 


Tasks  eliminated  x 

New  tasks  taken  on  x 

More  work  taken  on  x 

Sreater  range  of  information 
Greater  variety  of  services 
Fewer  mistakes 
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Better  forecasts  x 

Better  inage  x 

More  reliable  information  x 

More  creative  thinking  time  x 

Improved  turnround  time  x 

Quicker  production  tine  x 

Better  quality  output  x 

Improved  presentation  x 

Loner  staff  turnover  x 

Better  attendance  x 

More  job  satisfaction  x 

More  flexible  working  x 

More  sociable  hours  x 

Better  tools/facilitles/servlces  x 

Hands-on  experience  x 

Added  value  x 

Smoother  peaks/troughs  x 

Better  management  control  x 

Wider  spans  of  control  x 

Easier  communications  x 

Greater  organisational  flexibility  x 

Lesson  learning  x 

Flexibility  of  approach  to  calls  for  change  or  new  tasks  x 


There  is  an  understandable  tendency  to  overlook  some  relevant  advice  offered  in  the  early  1970s  (19): 

'Step  1:  Measure  whatever  can  be  easily  measured.  (That  is  O.K.  as  far  as  it  goes.) 

Step  2:  Disregard  that  which  cannot  be  measured,  or  give  it  an  arbitrary  quantitative  value. 
(This  is  misleading,  but  impressive.) 

Step  3;  Presume  what  cannot  be  measured  really  is  not  very  Important.  (This  is  blindness.) 

Step  4;  Conclude  that  what  cannot  be  easily  measured  really  does  not  exist.  (This  is  suicide.)' 

The  division  of  relevant  factors  according  to  their  ease  of  measurement  is,  nevertheless,  a  useful  step 
in  many  investigations.  A  current  study  at  Loughborough  has  examined  the  role  of  an  electronic  bulletin 
board  in  providing  a  focus  for  the  exchange  of  information  between  researchers  in  the  humanities.  The 
bulletin  board  was  set  up  on  JANET  (Joint  Academic  Network),  a  recently  established  network  connecting 
British  universities.  Its  operations  were  controlled  by  a  central  co-ordinator,  who  not  only  looked 
after  the  day-to-day  running,  but  also  carried  out  studies  of  the  way  the  network  was  used.  A  printed 
newsletter  was  run  in  parallel  with  the  bulletin  board,  so  that  coeparlsons  could  be  drawn  between  the 
two  media.  A  list  of  factors  we  would  like  to  know  about  was  drawn  up  and  divided  by  type.  Some 
examples  are  given  below. 

Quantitative  How  much  information  can  be  transmitted 

Speed  with  which  Information  can  be  transmitted 
Cost  of  transmission 
Reliability  of  transmission 

Qualitative  Timeliness  of  information 

Relevance  of  information 
Flexibility  of  system 
Accessibility  of  system 

The  conclusion  reached  from  this  study  was  that  a  list  containing  both  quantitative  and  qualitative 
factors  is  more  easily  evaluated  by  comparing  two  systems  (e.g.  printed  communication  versus  bulletin 
board)  than  by  trying  to  look  at  either  In  isolation.  Assessment  resolves  itself  into  asking  which 
system  is  preferable  under  each  heading.  After  each  characteristic  has  been  compared,  the  final  listing 
makes  it  relatively  easy  to  see  which  option  is  preferable  for  a  given  set  of  objectives. 

One  problem  in  obtaining  measurements  of  performance  relates  to  the  skills  needed  to  make  them.  Most 
factors  relating  to  economy  or  efficiency  lie  within  the  sphere  of  competence  of  information  centre 

staff.  But  effectiveness  measures  require  studies  of  consumer  reaction,  and  these  may  entail  skills 

that  members  of  an  information  centre  may  not  possess.  Us  r  studies  have  a  deceptively  simple  lo^ 
about  them,  but  can  be  difficult  to  carry  out  in  such  a  way  as  to  obtain  applicable  results.  Hence, 
given  the  need  to  emphasise  measurements  of  effectiveness  in  information  work,  it  may  prove  necessary 

either  to  provide  in-house  training,  or  to  hire  a  consultant.  An  exaaple  of  the  latter  approach  is 

provided  by  a  British  study  of  the  performance  of  public  libraries,  which  involved  consultancy  work  by 
the  National  Consimer  Council  (20).  The  report  emphasised  yet  again  the  need  for  clearly  defined 
objectives,  but  added  the  valuable  point  that  even  library  services  with  clearly  defined  objectives 
frequently  phrase  them  in  descriptive,  rather  than  prescriptive  terms,  so  making  the  step  to  performance 
measures  more  difficult.  All  customers.  It  Is  argued,  have  four  basic  questions  about  any  service; 

(1)  Does  a  product  or  service  do  what  it  is  supposed  to  do? 

(2)  Hhat  is  it  like  to  use? 

(3)  Mhat  does  it  cost? 

(4)  Does  it  do  what  It  is  not  supposed  to  do? 
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The  fine!  Item  U  not  usuelly  asked  -  at  least  In  this  form  -  In  information  work.  The  general  sector 
diagram,  us^  as  a  model  for  directing  consner  studies  in  general,  contains  an  interesting  elaboration 
of  these  questions  (see  below).  Though  some  of  the  factors  displayed  are  less  relevant  to  Information 
work,  it  is  useful  to  see  library  effectiveness  viewed  as  Just  one  facet  of  consumer  choice. 


This  leads  to  one  final  point.  Performance  measurement  is  linked  to  the  question  of  purpose;  for  whom 
are  the  measurements  being  made,  and  with  what  intention?  For  exairple.  data  are  often  needed  in  order 
to  convince  the  body  responsible  for  the  finance  of  the  centre  or  service  that  funding  should  be 
continued  or  even,  perhaps,  increased.  In  this  case,  the  need  is  for  performance  measures  which  will 
convince  the  funding  body  of  the  case  being  made.  Ways  of  providing  such  justification  might  be  listed 
under  the  following  headings  (21); 

(1)  Show  the  funder  details  of  how  the  resources  have  been  spent. 

(2)  Show  evidence  of  use  of  the  service. 

(3)  Find  the  cost  to  a  client  of  the  next  best  alternative  to  the  service,  and  show  that  It  is  greater 
than  the  cost  of  the  service. 

(4)  Find  what  clients  would  be  prepared  to  pay  for  the  service. 

(5)  Provide  evidence  of  benefit  gained  by  users  of  the  service. 

The  first  two  relate  more  to  Intermediate  output  measures  and  the  last  three  to  final  output  measures. 
But.  In  principle,  all  five  might  Involve  some  mix  of  measures  of  economy,  efficiency  and  effectiveness. 
What  typically  happens  is  that  the  provider  looks  for  the  sliaplest  mix  (in  terms  of  practical  measure¬ 
ment)  that  will  convince  the  funder.  Hence,  the  actual  set  of  performance  measures  to  be  Implemented  in 
this  sort  of  Instance  depends  on  the  particular  situation  and  people  Involved.  This  may  be  compared 
with  the  continuing  collection  of  standard  data  for  Intercoaparlson  between  institutions  and  for  long¬ 
term  trend  analysis. 

Huch  work  on  performance  measures  and  indicators  has  been  aimed  at  public,  national  and  academic 
libraries.  An  obvious  reason  is  because  these  categories  of  library  form  large  and  important  groupings 
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with  some  degree  of  public  eccountebility.  Special  libraries  and  information  services  tend  to  be  much 
more  disparate,  and  so  more  likely  to  set  up  their  own  variegated  criteria  for  performance  measurement. 
Hence,  more  systematised  work  still  remains  to  be  done  on  Information  centres  and  services,  but  they  can 
certainly  learn  from  all  the  developments  currently  under  way. 

PERFOMMCE  ICASURSCIfr  HMWALS 

The  most  useful  current  development  is  the  production  of  manuals  which  provide  sufficient  Instruction  to 
allow  the  systematic  collection  of  standard  data.  Two  recent  manuals  are  worth  considering  briefly  as 
examples  of  this  trend. 

The  first,  prepared  by  Line  and  published  by  UMESCO,  is  intended  for  international  use  in  measuring  the 
performance  of  dociaaent  supply  systems  (22).  There  is  a  detailed  discussion  of  the  factors  which  lead 
to  the  selection  of  the  specific  performance  measures  proposed  In  the  manual. 

(1)  The  fill  rate  -  the  proportion  of  document  reguests  that  are  fulfilled.  This  parameter  has  its 

limitations,  in  that  many  users'  needs  may  never  come  forward  as  requests.  In  addition,  what  is 

meant  by  'fulfilment'  may  be  ambiguous.  A  good  rate  of  fulfilment  in  one  subject  area  may  be  poor 
in  another.  Obtaining  the  information  about  the  doctaaent  can  also  be  more  than  a  simple  request: 
it  may  require  an  Iteration  between  the  information  service  and  the  requester  of  Information. 

(2)  Speed  with  which  a  request  is  fulfilled.  Since  fulfilment  entails  several  distinct  activities  - 

requesting,  processing,  supplying  -  these  may  need  to  be  examined  separately. 

(3)  User  satisfaction.  The  most  difficult  to  implement,  since  it  entails  questioning  the  users  them¬ 
selves. 

Various  measures  deemed  less  significant  are  also  discussed:  volume  of  demand  (particularly  changes  in 
it),  range  of  resources  available  (and  their  relevance  to  needsh  ability  to  accept  blind  requests 
(resources  to  be  extended  according  to  need),  robustness  and  flexibility  (ability  to  expand  and  adapt  to 
crises),  ease  of  use  (affected  by  the  simplicity  of  procedures),  quality,  minimal  loss  and  damage, 
balance  and  fair  distribution  of  supply,  capacity  to  serve  different  sectors  and  ease  of  monitoring. 
Line  explains  why  the  three  chosen  measures  are  basic  despite  their  limitations.  He  also  points  out  why 
any  such  measures  can  only  give  a  partial  picture.  Implicitly,  if  not  explicitly,  most  measures  are 
comparative.  So  it  is  usually  only  possible  to  say  that  one  service  is  doing  better  than  another,  or 
that  It  has  improved  its  own  performance  over  a  period  of  time.  Even  user  surveys  tend  to  be  compara¬ 
tive,  since  the  responses  depend  on  user  expectations  (which  may  be  low,  if  the  service  has  always  been 
poor).  In  any  case,  performance  measures  can  only  provide  part  of  the  Information  requires  in  order  to 
implement  change.  Looking  at  speed  of  supply,  for  example,  can  be  divided  into  three  steps  -  (1)  What 
is  the  present  speed?  (2)  where  do  delays  occur?  (3)  Hhat  are  the  causes  of  the  delay?  The  first  two 
steps  can  be  assessed  by  performance  measures;  but  the  third,  which  is  vital  in  deciding  what  changes  to 
introduce,  is  much  more  difficult  to  assess  in  this  way. 

The  discussion  of  the  items  Involved  Is  finally  translated  into  a  set  of  standard  forms  for  the  collec¬ 
tion  of  data  on  performance.  These  data  can  then  be  turned  into  the  appropriate  performance  measures. 
Two  levels  of  assessment  are  proposed.  The  first  leads  to  a  basic  set  of  performance  measures:  the 
amount  of  work  Involved  should  be  acceptable  even  to  a  small  Information  centre.  The  second  set  are 
more  sophisticated  and  would  provide  a  better  overview  for  larger  centres.  An  example  of  a  proposed 

form  for  collecting  basic  data  is  shown  below.  As  Line  stresses,  the  important  thing  is  to  standardise 

data  collection  in  this  way.  so  that  comparisons  can  be  carried  out  both  externally  and  internally. 

The  second  recent  manual  has  been  prepared  by  Hoore,  also  with  UNESCO  funding  (23).  It  is  concerned 
with  measuring  the  performance  of  public  libraries  on  an  international  basis.  Like  Line,  Noore  proposes 
there  should  be  two  levels  of  measurement  -  basic  and  more  sophisticated.  The  data  to  be  collected  and 
the  method  of  collection  are  examined  In  detail  under  three  headings  -  measures  of  resource,  measures  of 
output  and  consueer  opinion.  From  the  viewpoint  of  application  to  information  centres  and  services,  the 
last  discussion  is  particularly  useful.  At  the  basic  level,  questionnaires  or  interviews  can  be  used  to 
obtain  feedback  from  the  relevant  community  (i.e.  all  the  people  who  might  use  the  service)  or  the  users 
(l.e.  that  part  of  the  conminity  which  does  use  the  service).  At  a  more  sophisticated  level,  the 
suggested  methods  include  unobtrusive  testing,  where  researchers  put  themselves  in  the  position  of  users 
and  find  how  good  a  service  they  get;  attitude  surveys,  where  people's  attitudes  to  a  service  (rather 

than  their  use  of  It)  Is  explored;  group  discussions,  where  groups  of  people  (typically  users)  are 

brought  together  to  discuss  the  service;  depth  interviews,  where  individual's  use  of,  and  attitudes 
towards,  information  acquisition  are  examined  in  detail;  checklists,  which  can  be  used  for  a  quick  study 
of  specific  topics.  An  example  of  the  sort  of  basic  questionnaire  that  can  be  used  to  seek  user  feedback 
is  reproduced  below.  Like  Line,  Hoore  is  especially  concerned  with  the  standardisation  of  data  collec¬ 
tion.  His  manual  is  therefore  currently  in  draft  form  to  allow  the  incorporation  of  feedback  from 
attempts  In  various  countries  to  carry  out  the  recoamended  measurements. 

(mmsiOH 

How  and  why  measurements  of  performance  should  be  made  has  been  debated  extensively  for  the  past  two 
decades.  The  most  valuable  recent  development  has  been  the  growing  trend  towards  providing  standard 
methods  for  the  collection  and  analysis  of  performance  data.  At  present,  this  appears  to  be  further 
advanced  for  national,  public  and  academic  libraries,  than  for  special  libraries.  Individual  centres  or 
services  which  wish  to  measure  their  performance  systematically  can,  however,  glean  a  good  deal  of  value 
from  the  manuals  now  being  produced.  The  important  stage  remains  the  first  -  defining  what  it  is  wished 
to  achieve  in  terms  that  can  be  measured.  Moving  from  this  stage  to  the  actual  process  of  measurement 
can  often  be  facilitated  by  introducing  a  model.  So  the  stages  of  performance  measurement  may  be 
summarised  as: 

(1)  Define  objectives  (2)  Create  model  (3)  Define  measurements  (4)  Make  measurements  (5)  Analyse 
(6)  Implement. 
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Summary 

There  are  usually  three  viewpoints  from  which  an  information  system,  service  or 
product  can  be  evaluated.  The  first  is  that  of  the  system  or  service  producers, 
who  will  generally  be  assessing  the  extent  to  which  their  product  meets  its  assigned 
objectives,  and  who  will  be  particularly  concerned  with  the  improvement  of  efficiency 
and  effectiveness.  Such  evaluations  do  not  usually  involve  comparisons  with  competing 
services.  The  second  viewpoint  is  that  of  the  actual  or  potential  users,  whose 
concern  is  with  the  extent  to  which  a  system  or  service  meets  their  perceived  needs, 
and  its  related  cost.  This  assessment  often  involves  comparisons  between  services. 
The  third  viewpoint,  particularly  in  public-sector  operations,  is  that  of  the  system 
sponsors,  who  wish  to  assess  how  well  a  system  or  service  succeeds  in  meeting  policy 
objectives,  some  of  which  may  not  be  explicit. 

Quantitative  evaluative  data  are  often  hard  to  capture,  and  difficult  to  interpret. 
In  some  European  work,  a  more  subjective  approach  has  been  taken,  using  qualitative 
information  supported  by  such  numerical  data  as  may  be  gathered,  to  arrive  by  a 
judgmental  process  at  an  assessment  which  takes  account  of  the  political  factors 
inevitably  affecting  any  evaluation  other  than  the  most  basic. 

This  approach  is  illustrated  by  three  case  histories,  of  an  evaluation  of  services 
supplied  by  a  public  library  authority  to  local  government  officials,  of  the 
evaluation  of  the  ACRIS  service  produced  by  the  Food  and  Agriculture  Organisation 
of  the  United  Nations,  and  of  the  evaluative  processes  applied  to  proposals  for 
support  of  database  and  information  service  development  by  the  Commission  of  the 
European  Communities.  The  aim  is  to  illustrate  the  application  of  a  basic  philosophy 
of  evaluation  in  three  very  different  contexts. 


The  title  of  this  presentation  might  be  somewhat  misleading,  in  that  it  could  perhaps 
be  taken  to  imply  that  there  is  a  specifically  Curopean  approach  to  the  evaluallor 
of  information  systems  and  services,  as  opposed,  say,  to  the  American  school. 
This  is  definitely  not  the  impression  I  wish  to  give.  One  might  make  a  case  for 

the  existence  of  an  American  school,  on  the  grounds  that  most  of  the  'founding 
fathers'  of  practical  evaluation  work  in  our  field  -  Wessei,  Lancaster  and,  most 
significantly,  our  Course  Director  Don  King  -  are  American  and  have  done  most  of 

their  work  in  the  United  States,  but  European  workers  who  have  been  involved  in 

evaluation  work  respect,  admire  and  where  possible  copy  the  methods  used  by  our 
American  colleagues. 

That  is  to  say,  there  is  no  divergence  in  the  basic  philosophy,  or  the  mental 
attitude  that  one  brings  to  an  evaluation  exercise.  Where  there  are  differences 
in  the  work  done  in  America  and  in  Europe,  they  are  largely  attributable  to 

differences  in  the  individual  situations,  differences  in  what  is  being  evaluated, 
and  differences  in  the  reasons  for  doing  the  work  at  all.  As  far  as  methodological 
differences  are  concerned,  it  could  be  said  that  American  work  tends  to  be  based 
much  more  firmly  on  the  analysis  of  objective  data,  or  at  least  what  looks  like 
objective  data,  whereas  some  European  work  places  more  reliance  on  subjective 
assessment  and  individual  judgment,  supported,  wherever  possible,  by  numerical  data. 
The  difference  in  approach  is  to  some  extent  accidental,  but  very  s imp  1 i s 1 1 ca  1 1  y 
one  could  say  that  the  American  approach  tends  towards  the  quantitative,  and  the 
European  towards  the  qualitative.  Of  course,  personality  factors  affect  the  style 
of  work  that  is  done.  Evaluations  are  done  by  people,  and  the  leader  of  an  evaluation 
project  will  inevitably  adopt  the  style  of  work  that  he  finds  most  congenial,  and 
this  is  more  related  to  temperament  and  early  training  than  to  nationality. 

1  am  going  to  be  talking  about  some  studies  I  have  carried  out  or  been  involved 
in  myself,  and  I  shall  begin  by  saying  something  about  my  own  views  on  evaluation. 

Evaluation  is  generally  interpreted  as  meaning  the  measurement  or  estimation  of 
the  performance  or  other  characteristics  of  a  service,  with  a  view  to  determining 

the  extent  of  its  contribution  to  human  activities,  its  value  for  money,  its 

performance  in  relation  to  other  similar  services,  or  the  extent  to  which  it  has 
been  able,  or  is  likely  to  achieve  some  pre-speci f led  goals.  Sometimes,  as  in 

one  of  the  activities  I  shall  be  describing  later,  the  service  does  not  yet  exist, 

but  the  basic  principle,  of  trying  to  determine  'worth'  in  some  way,  still  obtains. 

Twenty  years  ago,  when  a  number  of  people  in  the  United  Kingdom  were  beginning  to 
be  interested  in  topics  like  'efficiency',  'effectiveness',  'cost-benefit'  and  so 
on,  all  concepts  that  relate  to  evaluation  in  some  way,  there  was  a  great  deal  of 
discussion  of  the  meaning  of  these  terms,  and  it  became  apparent  that  everyone 
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involved  in  the  activity  had  rather  different  definitions  of  these  terms.  My 

principal  contribution  to  the  discuasion  was  directed  towards  trying  to  encourage 
people,  especially  people  who  were  going  to  commission  work,  to  stop  using  the 

expression  'evaluation',  and  to  talk  instead  of  'assessment'  or  'appraisal'  or  some 
other  similar  term.  This  was  because  a  lot  of  people,  particularly  administrators, 
tended  to  think  of  'value'  as  being,  like  'cost',  measurable  in  money  terms,  so 
that  'evaluation',  meaning  'finding  out  the  value  of  something',  was  thought  to 
be  a  process  which  would  result  in  the  production  of  some  hard,  quantitative  and 
highly  meaningful  measures.  This  goal  is  often  unattainable  in  real  terms. 

Evaluation,  that  is  to  say,  has  a  number  of  related  meanings,  depending  on  the 
reasons  for  commissioning  an  evaluation,  who  commissions  it,  and  why.  There  is 

no  single,  absolute  method  of  evaluation,  and  any  service  can  be  evaluated  in  a 
number  of  ways,  from  a  number  of  different  viewpoints.  Some  of  the  criteria  used 
may  be  subjective,  some  objective,  and  some  apparently  objective  criteria  are  in 
fact  subjective.  'Assessment*  is  a  more  general  term,  which  is  less  of  an 
^  intellectual  strait- jacket  for  the  person  charged  with  carrying  out  the  work,  and 

which  conditions  the  sponsors  to  expect  a  less  severely  quantitative  result. 
Unfortunately,  to  politicians,  who  tend  to  think  in  extremely  simplistic  terms, 
'assessment'  is  a  less  exciting  term  than  'evaluation',  so  quite  often,  appraisals 
of  programmes  and  services  are  reported  as  'evaluations',  even  though  strictly 
speaking,  they  aren't. 

Broadly  speaking,  there  are  three  main  groups  from  whose  points  of  view  evaluations 
may  be  made.  These  are,  the  users  ur  potential  users  of  a  system,  the  managers 
and  operators  of  a  system,  and  the  political  or  other  sponsors  of  a  system. 

The  user  is,  in  general  terms,  concerned  with  getting  value  for  his  money. 
He  has  an  idea  of  what  he  wants  from  an  information  service,  and  is  concerned  with 
selecting  the  service  or  combination  of  services  which  most  closely  approximates 
to  his  ideal,  at  lowest  cost  to  himself  in  terms  of  money  and  effort.  User-oriented 
evaluation  often  entails  comparison  of  services  with  each  other.  Managers  and 
system  operators,  having  decided  the  characteristics  of  the  service  they  wish  to 
offer,  are  most  often  concerned  with  the  efficiency  aspects,  meaning  the  way  in 

which  the  service  product  can  be  achieved  with  minimum  use  of  resources.  The  system 
goals  towards  which  the  manager  works  are  usually  tactical.  System  sponsors,  on 
the  other  hand, are  generally  concerned  with  the  extent  to  which  strategic  goals 
are  achieved.  In  a  private  context  these  goals  are  related  to  maximisation  of 
revenue  or  profit,  or  market  share  achieved,  and  in  a  public  context,  to  the 
interrelationship  with  other  policy  objectives.  For  example,  production  of  a  guide 
to  environmental  information  resources  is  a  tactical  goal  or  objective,  from  the 
viewpoint  of  the  producers  of  the  guide;  from  the  sponsors'  point  of  view,  increasing 
the  level  of  environmental  awareness  is  a  strategic  goal,  to  which  the  production 
of  the  guide  is  a  tactical  contribution. 

Very  often,  the  user's  needs  or  desires,  or  the  system  producer's  goals,  ere  not 
sufficiently  explicitly  stated  as  to  be  able  to  provide  adequate  criteria  for  an 
evaluation.  In  such  cases,  the  evaluator  has  to  develop  or  adapt  the  criteria  to 

be  used,  on  the  basis  of  discussions  with  users,  managers  or  sponsors,  as  appropriate; 
study  of  previous  policy  statements  about  the  service  to  be  evaluated;  the  published 
prior  art;  and  his  own  experience.  Some  criteria  may  be  appropriate  to  more  than 
one  viewpoint,  or  to  more  than  one  type  of  system. 

I  now  want  to  spend  a  little  time  discussing  some  of  the  criteria  that  might  be 
appropriate  to  evaluations  from  the  three  viewpoints  I  have  identified. 

Use r -o r lent ed  criteria. 

The  user,  I  have  said,  ia  concerned  with  getting  value  ‘‘or  money,  a  vague  concept 
which  may  mean,  among  other  things,  maximising  the  number  of  requirements  satisfied 

while  minimising  the  money  or  effort  cost.  This  is  in  effect  a  cost-benefit  approach, 

getting  the  maximum  benefit  for  the  minimum  cost.  The  problem  here  is  usually  the 
defining  of  the  user  requirements  which  are  to  be  satisfied.  In  general  terms, 
a  user  is  normally  considered  to  want  all  information  relevant  to  an  enquiry  (in 
the  case  of  a  query-answering  service),  or  a  specific  answer  to  a  specific  question, 
or,  in  the  case  of  current  awareness  services,  all  the  current  information  relevant 
to  hia  job  or  professional  interests.  'AH'  information  may  be  modified  by  a  time 
constraint  -  'all  information  since  1980'  -  or  by  language  -  'all  in  English  or 

French'  -  or  by  some  other  consideration.  In  practice,  'all'  information  is  recog¬ 
nised  to  be  virtually  unobtainable,  and  the  average  user  will  settle  for  a 
subjectively-assessed  'most*.  The  degree  of  coverage  of  a  subject  by  a  particular 

.  service  can  be  aaaeased  in  a  variety  of  ways,  in  terms  for  instance  of  the  journal 

titles  covered  by  the  service,  or  more  elaborately  by  measuring  the  extent  of 
coverage  of  exhaustive  subject  bibliographies. 

Currency  of  response  is  a  factor  related  to  coverage,  and  the  currency  of  a  service, 
expressed  in  terms  of  the  time  interval  between  publication  of  a  piece  of  information 
and  ita  notification  by  the  service,  ia  measurable  reasonably  exactly  in  various 
ways. 


Both  coverage  and  timeliness  are  strictly  applicable  only  to  bibliographic  services, 
or  services  which  provide  access  to  a  constantly  incrementing  information  collection. 
A  bibliographic  service  is  covering  publications,  usually  but  not  exclusively  journal 
articles,  which  are  constantly  appearing,  and  each  issue  of  the  service  notifies 
a  new  time-slice  of  the  stream  of  publication.  Similarly,  a  numerical  database 
giving  information  on,  aay,  share  prices,  exchange  rates,  or  reports  from 
meteorological  stations  provides  a  stream  of  fresh  information.  A  numerical  data¬ 
bank,  such  as  an  index  to  low-resolution  spectra  of  inorganic  compounds,  or  data 
on  a  particular  family  of  organisms,  does  not  increment  in  the  same  way,  and 

timeliness  has  the  sense  of  'most  up-to-date  information'  in  a  slightly  different 
way,  while  coverage  ceases  to  apply,  except  in  the  sense  of  including  or  not  including 
the  members  of  an  essentially  finite  universe. 

Users  are  also  often  concerned  about  the  unit  amount  of  information  a  system  can 
present  to  them,  meaning,  in  a  bibliographic  system,  the  amount  of  information 

given  about  any  single  item  notified.  for  example,  in  a  titles-based  service,  is 

the  information  sufficient  to  identify  the  document  referred  to,  for  purposes  of 

acquisition^  In  an  abstracting  service,  does  the  abstract  allow  positive  decisions 
about  relevance;  does  it  serve  as  a  document  surrogate,  or  does  it  merely  indicate 

the  broad  subject?  Some  services  can  give  the  user  a  great  deal  mere  than  he  needs 

to  know,  and  this  must  be  taken  into  account.  For  example,  if  a  manufacturer  wants 
regularly  to  know  the  names  and  addresses  of  suppliers  of  particular  components, 
he  does  not  need  to  know  a  great  deal  about  the  financial  circumstances  or  the 
capitalisation  of  the  suppliers,  so  subscribing  to  too  elaborate  a  service  is 
only  increasing  his  costs  unproduct i ve 1 y . 

Ease  of  access  to  information,  either  in  t)road  terms  of  the  us  er  -  f  r  i  endlt  ne  ss 
of  the  svstem  or  the  simplicity  of  the  command  language,  or  in  terms  of  the  quality 
of  the  indexing,  is  also  important  to  users.  Much  of  the  early  effort  in  evaluation 
of  information  services  was  directed  towards  the  indexing  structure,  the  principal 
measures  used  being  recall,  the  ratio  between  the  number  of  relevant  documents 

retrieved  and  the  number  of  relevant  documents  in  the  system;  and  precision,  the 

ratio  between  the  number  of  relevant  documents  retrieved  and  the  total  number  of 

documents  retrieved.  Recall  and  precision  may  be  too  sophisticated  or  exotic  for 

8  user's  own  evaluation  of  services,  and  can  be  replaced  by  a  more  subjective 

assessment  of  general  service  usability.  Recall  and  precision  might  be  described 
as  confidence  factors,  because  they  relate  to  the  confidence  a  user  can  have  in 
the  system's  response  to  an  enquiry.  If  a  system  produces  no  relevant  references 
in  response  to  an  enquiry,  the  user  wants  to  know  with  what  confidence  he  can  then 

believe  that  nothing  has  been  published  on  the  subject  of  his  enquiry.  1  have, 
incidentally,  never  seen  any  tests  addressing  this  particular  point. 

Cost  is  an  important  aspect  of  service  use  from  the  user's  viewpoint.  Simple  cost 
of  service  alone  is  seldom  of  much  value  in  making  a  choice  between  services,  and 

a  more  meaningful  criterion,  for  a  bibliographic  service,  would  be  the  cost  per 

relevant  citation  retrieved,  or  per  useful  citation.  In  situations  where  more  than 
one  service  is  used,  it  is  sometimes  necessary  to  use  a  cost-per-citation  criterion, 
to  take  account  of  the  overlap  in  coverage  among  services.  Cost  should  include 
not  only  the  price  paid  for  a  service,  but  also  the  cost  of  labour  used  in  its 

consultation,  whether  that  labour  is  provided  by  the  user  himself  or  by  an  inter¬ 
mediary.  It  does  not  necessarily  follow  that  a  service  which  comes  out  well  on 
a  cost-per-citation  basis  is  necessarily  the  'best'  in  terms  of  satisfying  a  user's 
needs.  It  may  merely  mean  that  one  service  has  a  more  attractive  pricing  algorithm 
than  another,  is  mounted  on  a  cheaper  (possibly  subsidised)  host,  or  even,  to  take 
the  extreme  case,  has  very  few  useful  citations  at  all  but  the  few  there  are  are 
very  cheap  -  only  one  reference  in  the  system,  but  you  can  have  it  for  free. 

Assuming  there  were  a  way  of  quantifying  a  user’s  information  need,  a  more  reasonable 
criterion  would  be  related  to  the  unit  proportion  of  that  need  satisfied  per  unit 

cost . 

The  problem  is  further  complicated  in  an  economic  sense  by  the  fact  that  in  practice 
an  individual  user’s  resources  for  paying  for  information  (either  in  cash  or  in 
labour)  are  necessarily  finite,  and  frequently  small,  so  that  a  user  choice  may 
often  be  stated  as,  'How  much  can  I  get  for  my  money?',  or  'How  much  can  I  get  in 

half  an  hour'^'-  This  constraint  tends  to  enforce  a  qualitative  as  well  as  a 
quantitative  approach  to  system  products,  the  user  feeling,  for  example,  'From  this 
service  I  don't  get  many  references,  but  they  are  all  the  best  ones'.  This  factor 

may  underlie  the  success  of  services  based  on  the  contents  pages  of  the  best  or 

leading  journals,  and  ignoring  all  others. 

Many  evaluations  conducted  from  the  user's  viewpoint  involve  measuring  the 
performance  of  a  number  of  services  egainst  a  standard  set  of  criteria,  then,  by 
attaching  weights  to  the  respective  criteria,  determining  which  service  produces 
the  highest  rating.  Because  of  the  weights  attached  to  the  criteria  are  specific 
to  individual  users  or  user  groups,  this  is  an  effective  way  of  choosing  a  service 
most  likely  to  find  user  acceptance.  In  practice,  of  course,  few  users  undertake 

any  formal  or  systematic  evaluation  in  deciding  which  service  or  services  they 

intend  to  use,  preferring  to  operate  empirically  and  on  the  basis  of  subjective 

judgment.  As  has  been  pointed  out  from  time  to  time,  users  do  not  necessarily 
make  the  best  decisions,  they  make  the  easiest  ones. 
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Managerial  criteria. 

We  nay  now  consider  criteria  which  are  appropriate  to  system  evaluation  from  the 
viewpoint  of  designers,  nanagera  and  operators.  It  is  reasonable,  but  often 
unrealistic,  to  asaune  that  when  a  aysten  was  started,  a  number  of  objectives  were 
drawn  up,  such  that  the  extent  to  which  these  objectives  have  been  attained  can 
be  used  as  evaluation  criteria.  in  reality,  and  particularly  in  the  case  of  the 
older  abstracting  and  indexing  services,  systema  are  likely  to  have  evolved  from 
humble  beginnings,  often  in  an  academic  environment  or  from  learned  or  professional 
society  origins,  and  have  only  acquired  more  or  leas  explicit  goals  at  a  relatively 
late  stage  in  their  development. 

Bibliographic  services  usually  have  scope  definitions,  which  define  the  intended 
coverage  of  the  service,  in  terms  either  of  a  discipline  or  a  professional  grouping, 
or  a  range  of  journals  or  other  definable  class  of  literature.  If  coverage  is  defined 
in  terms  of  specific  jcurnal  titles,  then  there  is  usually  no  problem  in  determining 
the  degree  of  achievement  of  that  particular  goal.  If,  as  is  far  more  usual, 
definition  of  scope  is  in  terms  of  s  discipline  or  professional  group,  it  is 
usually  expressed  as  'the  literature  of  the  subject',  rather  than  'the  literature 
of  interest  to  workers  in  the  subject'.  Whereas  a  document  can  usually  be  defined 
as  being  about,  for  example,  chemistry,  the  literature  of  interest  to  chemists  can 

include  more  or  less  everything,  depending  on  the  professional  concerns  the 

individual  chemist,  and  the  literature  is  therefore  substantially  less  definable, 

locatable  and  collectable.  Implicit  or  explicit  value  judgements  often  qualify 
coverage  policies,  so  that  trivial  or  ephemeral  literature  is  excluded,  or, 

conversely,  only  the  research  or  scholarly  literature  is  included.  While  the 
information  needs  of  an  individual  user  can  often  be  relatively  firmly  defined, 
the  needs  of  whole  communities  of  users  are  obviously  much  harder  to  specify. 
Consequently,  it  is  in  general  easier  to  evaluate  the  performance  of  a  service  :n 

relation  to  a  specific  user  than  it  is  to  evaluate  it  in  relation  to  the  managerial 
goals. 

Timeliness  or  currency  are  important  to  system  managers,  as  they  are  to  users,  more 
so  in  the  case  of  services  with  a  large  current-awareness  component.  Services  which 
see  their  role  as  primarily  or  partly  archival,  or  which  are  intended  principally 
for  retrospective  search,  may  give  less  weight  to  timeliness.  Recall  and  precision, 
the  tests  of  effectiveness  of  the  indexing  structure,  are  of  concern  also,  as  they 
are  to  the  user. 

Many  of  the  criteria  of  importance  to  system  operators  are  managerial  in  nature, 
concerned  with  the  clerical  or  machine  processes  of  compilation  of  a  database,  and 
so  are  oriented  towards  cost-effectiveness,  whereas  user  evaluation  has  more  of 
a  coat-benefit  style.  Quality  control  aspects,  including  conformity  to  agreed 
standards,  error  reduction  and  avoidance  of  duplicate  entries  make  up  much  of  the 
managerial  concern  beyond  the  cost-effectiveness  level. 

In  considering  questions  of  quality,  some  consultation  and  study  of  the  potential 
or  actual  users  may  be  implemented,  although  a  service  as  provided  is  usually  a 
compromise  between  the  user’s  notion  of  an  ideal  service  and  the  operator's  perception 
of  what  is  technically  feasible  at  an  economic  price.  Included  in  managerial 
evaluation  are  questions  related  to  the  possibility  of  producing  the  same  system 
outputs  at  reduced  cost,  by  cheaper  input  processes,  or  by  automation  of  some  or 
all  activities,  by  reductions  or  restructuring  of  staff,  or  by  other  means.  The 
coating  aspects  of  a  managerial  evaluation  are  often  complicated,  in  relatijn  to 
the  production  of  a  particular  service,  by  some  shared  processing  with  another 
service;  input  acquired  may  be  used  in  producing  more  than  one  service,  so  that 
the  cost  of  acquisition  may  have  to  be  treated  as  an  overhead  and  partitioned  among 
several  services,  usually  on  a  relatively  arbitrary  basis. 


Sponsor  criteria. 

Lancaster  and  Climenson  (1),  in  an  early  paper  on  evaluating  the  economic  efficiency 
of  document  retrieval  systems,  noted  that  the  interest  of  the  system  sponsors  (or 

in  their  terminology,  top  management)  'is  concerned  with  a  higher  level  of  evaluation, 
namely  assessment  of  the  worth  of  the  system  or  the  economic  justification  for  its 
existence'.  They  suggest,  rightly*  that  this  is  the  hardest  level  at  which  to 

conduct  an  evaluation.  I  would  add  that  in  many  cases,  the  sponsors  are  concerned 
not  only  with  the  economic  justification  of  the  system  or  service,  but  also  with 
its  political  justification.  It  is  simplistic  to  suppose  that  straightforward 
prof it-and- loss  considerations  are  all  that  is  taken  into  account  when  sponsors 

decide  whether  to  initiate,  continue,  change  or  abandon  information  systems  or 

services,  although  of  course  in  some  cases,  the  economic  factors  are  the  prime 

consideration. 

In  the  simple  case,  and  in  a  purely  commercial  setting,  if  a  system  makes  a  profit 
then  it  is,  on  the  face  of  it,  successful,  but  even  this  simple  measure  of  success 
can  be  hard  to  determine.  What,  for  instance,  ia  a  'profit'?  Is  it  an  excess 

of  annual  revenue  over  annual  running  cost,  or  are  the  development  coats,  which 
are  often  vry  high,  to  be  taken  into  account?  In  a  business  context,  one  is 

concerned  with  the  return  on  investment,  and  therefore  it  is  necessary  to  consider 


the  total  development  and  marketing  costa  as  vrell  as  operating  costs,  and  to  be 
able  to  assess  at  what  point  the  initial  investments  will  be  repaid.  Even  if  the 

full  costings  look  encouraging,  so  that  it  is  possible  to  say  that  a  particular 
service  is  successful  in  market  terms,  and  the  financial  backers  are  satisfied, 
this  does  not  give  much  indication  of  the  value  placed  on  the  service  by  its  clients, 
because  there  is  always  the  possibility  that  it  is  underpriced. 

An  apparently  unprofitable  service  can  be  supported  even  in  a  commercial  environment, 
for  a  number  of  reasons  connected  with  the  policy  of  the  producers.  An  unprofitable 
online  database  may  be  carried  by  a  host  because  it  fits  in  well  with  the  other 
databases  carried  on  the  system,  giving  a  more  rounded  'product  mix'.  This  is 
similar  to  the  way  in  which  publishers  will  produce  some  prestige  titles  although 

they  know  they  will  not  be  very  profitable,  because  it  enhances  the  reputation  or 

the  image  of  the  publishing  house.  A  service  might  also  be  carried  because  it  is 

a  first  attempt  to  enter  a  new  market,  and  the  producer  or  the  host  wishes  to  show 
some  interest  or  expertise  in  a  particular  product  area.  If  any  of  these 
considerations  obtain,  the  evaluator  must  know  about  them,  and  be  able  to  say 
something  about  the  extent  to  which  the  service  under  consideration  is  meeting 
these  rather  nebulous  objectives. 

When  the  sponsors  are  governmental  or  international  bodies,  a  great  many  other 
considerations  are  likely  to  apply,  many  of  which  will  not  be  very  explicit. 
Obviously,  the  cost  *e  f  f  ect  i  veness  of  the  system  or  service  will  be  of  almost  as 
much  interest  as  it  would  be  to  a  commercial  sponsor,  because  public  bodies  are 
accountable  for  the  expenditure  of  public  funds.  However,  as  public  sector  services 

are  often  offered  at  low  or  no  cost  to  their  users,  the  concept  of  profit  will 

seldom  apply.  In  some  cases,  a  public  sector  service  may  be  offered  because  it 

is  a  legal  requirement  that  information  should  be  made  available  to  the  public, 

and  it  may  be  necessary  to  maintain  a  service  even  if  its  use  is  very  small,  because 

of  that  nece  ity.  In  some  other  cases,  the  public  sector  may  maintain  a  service 

even  if  no  statutory  requirement  exists,  because  it  considers  the  service  to  be 

necessary  in  the  public  interest.  Here  again,  economic  factors  may  not  be  very 

important,  although  it  is  legitimate  for  the  evaluator  to  comment  on  the  nature 
of  the  public  interest  which  is  being  served,  and  the  extent  to  which  public 
interest  could  be  met,  by  other  means. 

Some  services  may  be  made  available  because  it  is  considered  that  the  public  sector 
can  maintain  a  more  neutral  stance  vis-a-vis  the  information  offered  than  would 
perhaps  be  the  case  with  a  commercial  operator.  For  instance,  the  European  Commission 
offers  services  giving  guidance  on  database  selection,  considering  that  it  can  be 
more  objective  than  any  individual  host  or  database  provider,  because  it  has  no 
financial  stake  in  the  market.  * 

Public  sector  bodies  are  as  concerned  about  their  public  image  as  are  commercial 
concerns,  and  in  looking  at  services  produced  by  governmental  and  international 
organisations,  not  only  the  information-related  aspects  of  service  operation  should 
be  considered,  but  also  the  extent  to  which  the  service  offered  contributes  to  the 
visibility  of  the  sponsoring  organisation,  or  serves  to  demonstrate  its  interest 
and  concern  in  a  particular  area. 

There  may  be  many  other  related  goals  to  consider.  For  instence,  a  part  of  the 
overall  policy  of  the  European  Commission  is  to  reduce  regional  disparities  between 
the  Member  States.  In  an  information  context,  this  means  that  in  an  ideal  world, 
Member  States  in  which  the  information  industry  and  the  information  infrastructure 
ve  weak  should  be  supported  and  encouraged  to  develop  themselves  to  the  same  level 
of  information  activity  and  ability  as  the  more  informationally  advanced  Member 
States.  One  way  of  doing  this  might  be  to  support  systems  which  allow  the  development 
of  expertise  and  technical  capability,  or  which  encourage  infrastructural  development. 
Even  though  such  systems  may  not  be  profitable  in  cash  terms,  or  may  not  be  serving 
very  large  user  communities,  the  extent  to  which  they  contribute  to  the  achievement 
of  the  broader  political  goals  may  often  justify  the  investment  of  public  funds. 
The  evaluator,  therefore,  must  be  aware  of  these  political  goals,  and  must  endeavour 
to  measure  the  system  or  service  impact  in  relation  to  them. 


Common  aspects  of  evaluation  at  different  levels. 

There  are  some  features  which  are  common  to  all  levels  and  types  of  evaluation. 
The  two  most  important,  which  are  to  some  extent  inter- re  la ted ,  are  objectivity 
and  comprehensibility.  They  are  equally  important. 

It  would  be  taken  for  granted  that  a  sound  evaluation  s.iould  be  completely  objective. 
This  means  that  not  only  is  the  person  or  team  conducting  the  evaluation  free  from 
bias  or  preconceptions  about  what  is  being  evaluated,  but  also  the  methodology  used 
should  as  far  as  possible  be  such  as  not  to  introduce  any  bias  into  the  results. 
If  a  person  who  is  responsible  for  the  operation  of  a  system  is  also  the  evaluator, 
it  is  possible  that  the  evaluation  would  be  done  without  bias,  but  it  is  not  very 
likely,  and  what  is  more  important,  many  people  would  not  believe  it  to  be  free 
from  bias.  Not  only  must  any  evaluation  be  impartial,  it  must  be  seen  to  be 
impartial,  and  this  is  particularly  true  whenever  political  factors  have  to  be  taken 
into  account.  Visible  objectivity  is  also  essential  in  order  to  secure  the  co¬ 
operation  of  the  system  or  service  to  be  evaluated. 
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No  individual  is  completely  impartial  about  everything,  but  as  far  as  possible  the 
evaluators  should  be  without  vested  interest  in  the  operation,  the  subject  matter 
or  the  financial  support  of  whatever  system  or  service  is  being  evaluated. 

In  order  to  guard  against  possibilities  of  prejudice  or  partiality,  in  situations 

where  everyone  may  be  assumed  to  be  notionally  predisposed  one  way  or  the  other, 
then  it  is  desirable  to  have  the  evaluation  carried  out  by  a  team  so  constituted 

that  its  various  prejudices  may  cancel  each  other  out.  If  evaluating  an  international 
system,  for  instance,  in  order  to  smooth  out  political  and  national  differences 

it  would  be  normal  to  use  a  team  with  representatives  from  the  Western  democracies, 

the  Soviet  bloc  and  the  Third  World. 

By  comprehensibility,  1  mean  that  whoever  has  commissioned  the  evaluation  should 

understand  what  is  going  to  be  done  in  the  course  of  its  conduct,  and  should  also 

be  able  to  understand  what  the  findings  mean,  and  what  are  their  implications  for 
management  or  policy.  This  means  that  at  the  outset  of  an  evaluation  exercise, 
the  aspects  to  be  examined  should  be  discussed  with  and  agreed  by  those  who  are 
paying  for  the  exorcise,  and  if  possible  by  the  operators  of  the  system  or  service 
to  be  studied.  System  goals  are  often  couched  in  vague  or  general  terms,  not 
sufficiently  precise  to  serve  as  criteria  for  evaluation,  and  these  goals  need  to 
be  made  as  precise  as  possible. 

The  methodology  to  be  used  is  properly  the  concern  of  the  evaluator,  who,  after 

all,  is  or  should  be  the  expert,  but  it  is  useful  to  explain  what  is  going  to  be 

done,  and  what  sort  of  results  can  be  expected.  Particularly  in  more  political 

settings,  it  is  very  important  that  the  broad  shape  of  the  final  report  should  be 
agreed  before  any  work  is  started.  A  certain  amount  of  care  has  to  be  exercised 

in  discussing  the  methodology,  because  everybody  with  any  connections  with  the 

information  world  is  an  amateur  evaluator  at  heart,  and  it  can  sometimes  '.>e  very 

difficult  to  fend  off  the  helpful  suggestions  and  advice  that  may  be  offered. 

It  may  be  necessary  sometimes  to  suffer  the  support  of  an  advisory  committee,  but 
the  person  actually  responsible  for  the  conduct  of  an  evaluation  should  always  make 
it  clear  that  he  or  she  is  in  charge.  You  must  have  confidence  in  your  own  expertise 
and,  while  remaining  open  to  genuinely  helpful  suggestions,  never  let  a  committee 
design  your  questionnaires  for  vou. 

When  a  study  has  been  carried  out,  its  results  and  findings  will  be  communicated 

in  a  report  to  those  who  commissioned  the  study.  Whatever  the  findings  may  be, 

they  must  be  explained  in  a  way  which  is  unamb i guous 1 y  understandable  by  the 
recipients.  However  erudite  the  methods  and  measures  are  that  have  been  used, 
they  must  be  translated  into  plain  terms,  because  the  object  of  the  exercise  is 
to  assist  someone  to  take  a  decision.  It  is  not  the  role  of  the  evaluator  to  take 
the  decision  himself,  unless  recommendat ions  of  that  kind  have  been  specifically 
called  for,  but  it  is  his  duty  to  provide  the  material  to  inform  dec  i  s  j  on-mak  i  ng , 

in  such  a  way  that  it  can  be  directly  used.  1  think  there  is  often  a  duty  on  the 

evaluator  to  go  a  little  further  than  simply  presenting  the  findings.  In  many  cases, 
those  who  commission  evaluations  are  not  themselves  a  part  of  the  information  world, 
although  it  is  to  be  hoped  that  they  have  a  broad  general  understanding  of  it. 

It  is  therefore  desirable  that  as  well  as  presenting  the  findings  of  an  enquiry, 
some  sort  of  attempt  be  made  to  interpret  the  findings,  by  giving  some  discussion 
as  to  why  the  findings  are  the  way  they  are.  One  of  the  things  that  should  come 
out  of  a  good  evaluation  is  some  degree  of  insight  into  the  way  information  systems 
and  services  work,  and  the  relationship  between  information  and  the  informed. 
An  evaluator  is,  in  the  course  of  his  work,  constantly  improving  and  adding  to  his 
insights  into  the  information  situation,  and  it  is  his  duty  to  share  those  insights. 
After  reading  the  report  on  an  evaluation,  the  reader  should  feel  that  he  knows 
more  not  only  about  the  object  of  direct  study,  but  about  the  information  process 
as  a  whole. 

It  is,  of  course,  essential  that  the  reader  can  distinguish  between  conclusions 
which  are  supported  by  the  data  and  observations  collected  in  the  course  of  the 
study,  and  those  which  are  supported  by  no  more  than  the  imagination  and  the 
experience  of  the  author. 


Evaluation  of  a  local  information  service. 

Now  I  want  to  move  to  some  descriptions  of  evaluative  studies  in  which  I  have  been 
involved,  to  illustrate  some  of  the  points  1  have  been  making,  and  to  show  in  some 
cases  how  a  relatively  small  amount  of  hard  numerical  data  is  fleshed  out  by 
qualitative  information  and  judgment. 

The  first  study  I  want  to  describe  was  carried  out  in  1980,  in  the  county  of 
Leicestershire  in  the  United  Kingdom  (2).  Leicestershire,  like  every  county  in 
the  UK,  has  a  public  library  system  which  is  funded  and  administered  by  the  local 
authority.  Speaking  from  memory,  in  1980  the  Leicestershire  Library  Authority 
budget  was  of  the  order  of  four  and  a  half  million  pounds,  and  it  operated  about 
seventy-five  main  and  branch  libraries  within  its  area.  The  principal  roles  of 
the  public  library  system  are  given  in  the  Act  of  Parliament  (3)  which  establishes 
them  as  'to  provide  a  comprehensive  and  efficient  library  service  for  all  persons 
desiring  to  make  use  thereof*.  This  means,  in  effect,  that  they  lend  books  and 
other  materials  free  of  charge,  and  provide  reference  service,  to  members  of  the 
general  public.  Many  of  the  public  library  authorities  do  much  more  than  this, 
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and  as  an  example  Leicestershire  at  the  time  of  which  1  am  speaking  was  providing 
a  specialised  service  to  local  government  officers  and  others,  comprising  a  series 
of  current  awareness  bulletins,  a  photocopy  service,  a  query-answering  service, 
and  document  request  and  delivery  facilities. 

The  current  awareness  bulletins,  which  were  the  principal  ( jcus  of  the  evaluation, 
included  five  regular  bulletins.  One,  called  'Daily  Scan',  was  a  daily  survey  of 
the  major  newspapers,  abstracting  news  and  other  items  appearing  in  the  major 
national  daily  papers  and  the  Sunday  press,  plus  the  day's  Parliamentary  activities 
and  relevant  radio  and  television  programmes.  'Local  Scan'  was  produced  once  a 
week,  and  provided  similar  coverage  of  the  local  newspapers.  'Scan  in  parliament’ 
was  produced  twice  a  week  when  Parliament  was  in  session,  and  abstracted  debates 
in  the  Lords  and  Commons,  standing  committees,  and  recent  official  and  political 

publications.  'Scan  Review*  was  a  weekly  abstracts  bulletin  covering  material 
relevant  to  local  government  drawn  from  about  600  periodicals.  'Social  Work 
Information  Bulletin'  was  a  fortnightly  abstracting  service  of  recent  periodical 
literature  covering  all  aspects  of  social  work. 

When  the  study  I  am  describing  was  first  suggested,  there  were  three  reasons  why 
the  Leicestershire  library  service  felt  that  an  evaluation  was  necessary. 
Firstly,  the  library  service  itself  wanted  to  be  assured  that  provision  of  the 
services  was  a  wise  use  of  its  limited  resources.  Secondly,  the  library  authority 
felt  the  need  to  be  able  to  demonstrate  to  its  masters,  the  County  Council,  that 
the  perceived  benefits  in  relation  to  the  work  of  the  recipients  of  the  services 

justified  the  costs  of  providing  the  services.  There  was  also  a  desire  to  be  able 
to  assist  other  library  authorities  who  wished  to  initiate  similar  services,  by 
giving  them  guidance  on  the  levels  of  cost  involved  and  the  perceived  benefits  that 
could  be  expected.  After  the  General  Election  of  May  1979,  greater  emphasis  was 

placed  on  containing  and  reducing  the  levels  of  public  spending,  and  this  made  the 

first  two  of  these  needs  very  much  more  acute.  The  pressures  on  local  authority 
budgets  meant  that  there  was  no  certainty  that  the  services  could  be  kept  in  being 
without  change  for  any  length  of  time,  and  information  about  their  utility  was  needed 
very  rapidly,  in  order  to  take  decisions  about  their  future.  This  meant  that  any 
evaluation  mounted  had  to  be  carried  out  as  rapidly  as  possible. 

In  passing,  this  sort  of  situation  occurs  fairly  frequently.  In  my  experience, 
most  evaluations  have  to  be  carried  out  in  as  short  a  time  as  possible,  as  soon 
as  possible,  producing  as  much  information  as  possible,  and  on  as  low  a  budget  as 
possible.  And  the  harder  the  problems  are,  the  less  is  the  time  available  for  their 
solution. 

The  Scan  bulletins,  with  the  exception  of  the  Social  Work  one,  were  not  designed 
to  serve  the  needs  of  a  single  usei  except  m  so  tar  as  Local  Government 

officers  of  all  departments  and  disciplines  can  be  said  to  constitute  a  group,  nor 
were  they  intended  to  meet  the  full  range  of  their  recipients'  information  needs. 

Instead,  they  were  designed  to  provide  coverage  of  potentially  relevant  information 
appearing  in  specific  publications,  implicitly  being  intended  to  save  the  users  the 
effort  of  scanning  those  publications  themselves  in  order  to  maintain  their  state 
of  awareness.  In  these  circumstances,  rather  than  adopting  a  methodology  designed 
to  determine  whether  and  to  what  extent  a  particular  set  of  notional  benefits  was 

realised,  it  seemed  better  to  take  an  approach  which  would  indicate  what  benefits 
from  the  use  of  the  bulletins  were  realised  by  the  user  population.  Matching  service 
provision  against  a  predetermined  set  of  user  needs  would  have  been  a  more  rigorous 
approach,  but  it  was  felt  that  the  more  open-ended  method  would  be  more  likely, 

given  the  variety  of  user  groups  and  the  likely  variation  in  their  information  needs, 
to  provide  more  insight  into  the  uses  and  perceived  values  of  the  services. 

The  study  was  therefore  designed  in  three  parts.  The  first  was  based  on  semi- 
structured  inteiviews  with  about  fifty  of  the  users  of  the  service.  The  second 

part  involved  a  very  brief  questionnaire  distributed  to  the  majority  of  users,  and 

the  third  was  an  examination  of  the  costs  and  effort  involved  in  production  of  the 

services.  About  five  hundred  copies  of  each  issue  of  each  of  the  services  we/e  printed 
and  distributed  (except  the  Social  Work  bulletin,  of  which  more  than  nine  hundred 

copies  were  distributed),  but  many  copies  were  seen  by  more  than  one  person,  so 
the  total  number  of  users  was  somewhat  indeterminate. 

Interviewees  were  selected  to  provide  a  representative  cross-section  of  departmental 
staffs  within  the  Local  Government  structure,  together  with  representatives  of  the 
County  Council,  the  City  Council,  the  Leicestershire  Constabulary  and  the  Area  Health 
Author! t  y . 

The  purpose  of  the  interviews  was  to  gather  information  on  the  uses  of  the  bulletins; 
their  contribution  to  professional  and  general  awareness;  any  deficiencies  of  the 
services  or  difficulties  with  their  format,  presentation  and  coverage;  estimates 
of  efforts  saved  by  their  use,  or  replacement  effort  which  would  have  to  be  put 
in  if  the  services  became  unavailable;  and  some  feelings  for  the  notional  value, 
in  cash  terms,  of  the  services.  An  interview  schedule  was  drawn  up,  with  forty- 
one  questions  covering  the  areas  mentioned,  but  this  was  intended  to  be  no  more 
than  an  aide-memoire.  The  interviews  took  between  an  hour  and  an  hour  and  a  half 
eac’i.  Interviews  opened  with  a  brief  statement  by  the  interviewer  (myself) 
about  the  reasons  for  the  interview,  and  the  funding  for  the  study,  to  make  it  clear 
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that  the  interviewer  was  independent  of  the  Leicestershire  administration  and  the 
library*  and  that  Leicestershire  was  not  paying  for  the  investigation  (it  was  in 
fact  paid  for  by  the  British  Library).  Following  this,  the  general  pattern  of  the 
interview  took  the  shape  of  a  conversation,  in  the  course  of  which  responses  to 
the  points  on  the  interview  schedule  were  obtained,  not  always  in  the  order  in  which 
they  appeared  there.  Because  of  the  conversat iona 2  style,  both  interviewer  and 
interviewee  felt  at  ease  with  each  other,  and  the  interviews  were  generally  felt 
to  be  agreeable  experiences  for  both  participants.  This  was  a  little  unexpected, 
but  gratifiyng,  and  it  certainly  encouraged  a  free  flow  of  useful  information 
from  the  respondents.  It  should  be  noted  that  this  can  sometimes  be  dangerous, 
because  the  interviewees  may  be  telling  you  what  they  think  you  would  like  to  hear, 
rather  than  what  they  actually  feel,  but  in  this  case,  the  professional  attitude 
of  the  interviewees  and  the  frankness  with  which  they  expressed  their  views 
encouraged  me  to  think  that  this  was  not  happening. 

The  interviews  provided  a  great  deal  of  information,  and  at  the  end  of  the  senes 
I  felt  that  I  knew  much  more  than  before  what  it  was  like  to  be  a  local  government 
official,  what  sort  of  work  they  did,  the  different  types  ot  information  they  needed 
and  used,  and  ha# they  organised  tieir  professional  lives.  This  helped  to  form 
subjective  judgments  about  the  location  of  the  Scan  services  in  the  spectrum  of 
their  information-acquiring  activities,  and  in  effect  set  the  services  context 

in  a  way  that  might  not  have  been  possible  had  a  more  formal  and  detached  user  study 
methodology  been  adopted. 

Some  of  the  information  gained  was  what  I  would  class  as  ’  quas i -quant i ta t i ve ' ,  in 
the  sense  that  I  was  able  to  count  the  frequencies  with  which  different  responses 
were  given,  as  would  have  been  the  case  had  I  been  using  a  formal  survey  approach. 
I  use  the  term  ' quas i -quant  i  t at  i  ve '  to  draw  attention  to  the  dangers  of  assuming 
that,  because  one  has  been  able  to  produce  a  number  in  the  course  of  an  evaluative 
study,  the  number  necessarily  means  something  concrete.  for  instance,  thirty-seven 
of  the  people  interviewed  said  that  they  habitually  took  professional  reading  home 
with  them,  and  four  said  they  occasionally  did  so.  To  say  that  lb.b%  of  the 

respondents  habitually  took  professional  reading  home  would  be  accurate,  but  not 
particularly  helpful.  To  extrapolate  and  say  that  of  all  local  government 

officers  do  so  would  of  course  be  completely  unjustified.  Using  percentages  is 
a  device  often  used  by  consultants  to  draw  attention  away  from  the  fact  that  they 
have  used  very  small  samples,  and  often  lends  a  spurious  air  of  precision  to 

essentially  imprecise  data.  The  proper  conclusion  to  draw  from  a  finding  of  this 
kind  is  that  on  the  basis  of  a  fairly  small  but  reasonably  representative  sample, 
it  looks  as  though  about  three  out  of  four  local  government  officials  in  this 
particular  authority  take  professional  reading  home  with  them.  (It  does  not 

necessarily  follow  that  they  then  read  it,  so  that  point  should  be  explored  further). 

The  issue  of  value,  expressed  in  money  terras,  is  often  central  to  an  evaluation 
exercise,  but  it  is  an  extremely  difficult  area  to  explore.  There  are  many 
different  approaches.  One  method  is  to  try  to  estimate  the  cost  in  time  and  money 
that  would  be  incurred  by  the  user  if  he  had  to  achieve  the  same  level  of  informedness 
without  the  aid  of  the  service  under  study.  In  the  case  of  the  Daily  Scan,  this 

would  have  meant  estimating  the  time  that  would  have  had  to  be  spent  by  the  individual 
users  in  looking  quickly  th**ough  the  daily  national  press  to  provide  themselves 
with  the  same  amount  of  information  that  was  delivered  by  the  service;  strictly, 
to  find  the  same  items  for  themselves  that  they  had  identified  in  the  service  as 
being  relevant.  This  cost  is  then  set  against  the  unit  cost  of  the  service,  plus 

the  Cost  of  time  spent  using  it,  and  the  relationship  between  the  two  sets  of  costings 

is  used  to  get  an  estimate  of  the  value  of  the  service  to  the  individual.  This 

is  quite  a  good  approach,  but  it  reduces  the  role  of  an  information  service  to  that 
of  being  simply  a  timesaver.  In  real  life,  many  information  service  users  would, 

if  the  service  became  unavailable,  simply  be  less  well  informed.  In  Leicestershire, 
if  the  Daily  Scan  had  been  unavailable,  virtually  none  of  the  users  would  have 

attempted  to  scan  the  whole  range  of  the  daily  newspapers  for  themselves,  but  would 
have  made  do  with  one  national  paper  and  one  local.  The  'timesaver’  approach,  that 
is  to  33y»  does  not  take  account  of  the  value  of  being  better  informed. 

In  the  Leicestershire  study,  partly  because  of  reservations  about  the  n.ethodoi''qy 
and  partly  because  of  the  time  constraints,  the  'time  saved'  approach  was  not  used, 
although  what  sort  of  replacement  activity  would  be  undertaken  if  the  services  were 
withdrawn  was  explored  in  the  interviews.  To  get  an  idea  of  the  perceived  value, 
I  asked  the  Interviewees,  'If  this  were  a  commercial  service  bought  in  from  outside, 
like  a  journal  or  magazine,  what  sort  of  price  tag  would  you  expect  to  see  it 
carrying?'.  The  question  was  put  this  way  to  get  an  :dea  of  perceived  value  as 
reflected  by  expected  cost,  and  to  avoid  asking  directly  'How  much  would  you  pay 
for  it?'.  Most  of  the  respondents  answered  by  attempting  to  calculate  a  notional 

cost  of  production  and  guessing  the  circulation,  often  quite  accurately. 
I  then  asked  them  whether  they  would  authorise  the  purchase  of  the  service  from 

their  budgets,  at  the  cost  they  had  estimated.  All  except  five  of  the  interviewees 
said  they  would.  As  you  will  realise,  to  dress  this  up  as  some  sort  of  hard 
estimate  of  value  would  be  un  justi  f  i  but  at  least  it  showed  that  most  of  the 

respondents  did  attach  some  perceived  value  to  the  service. 

Qualitatively,  the  interviews  produced  ample  evidence  that  as  a  result  of  receiving 
the  services,  the  interviewees  certainly  felt  they  were  better  informed  than  they 
had  been  before  the  services  were  introduced.  Regular  patterns  of  use  of  the 
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services  had  been  formed,  quite  a  valuable  finding  because  if  use  of  an  information 
service  has  been  built  into  the  work  habits  of  the  users,  it  is  an  indication  that 

they  consider  it  to  be  valuable  to  them.  There  were  many  comments  about  the  style 

and  format  of  the  bulletins,  in  general  reflecting  a  satisfactory  level  of  confidence 
in  the  ability  of  the  producers  to  select  and  index  the  material  presetited. 

One  interesting  finding,  which  was  not  specifically  sought,  but  which  indicated 
a  political  advantage  gained  by  the  library  administration  in  producing  the 
bulletins,  was  that  regular  receipt  of  the  service  had  made  many  of  the  recipients 
more  aware  of  the  existence  and  capabilities  of  the  library  service  generally,  to 
the  extent  that  a  number  were  making  more  active  use  of  the  library  than  they  had 
in  the  past.  This  could  not  be  classed  as  a  direct  benefit  to  the  library,  because 
it  had  not  been  one  of  the  goals  of  the  library  service  in  establishing  the  Scan 

bulletins,  but  it  could  be  taken  to  be  a  spin-off  benefit  to  the  users,  in  that 
their  general  information  awareness  and  specific  knowledge  of  the  services  available 
to  them  had  been  increased. 

The  second  part  of  the  study  was  more  productive  of  quantitative  information. 
It  consisted  of  the  distribution,  with  every  copy  of  one  issue  of  each  bulletin, 
of  a  brief  questionnaire.  Overall,  about  of  the  distributed  questionnaires 

were  returned,  which  is  not  a  bad  response  for  this  type  of  survey,  given  no  follow¬ 
up  or  reminder  procedures.  There  were  only  four  questions,  two  of  which  were 
directed  to  estimating  the  number  ot  people  who  saw  the  services,  or  on  whose  behalf 
they  were  used.  These  questions  produced  reasonably  hard  figures,  indicating  that, 
for  example,  the  Daily  Scan,  of  which  some  500  copies  were  printed,  was  seen  by 

about  1300  people  and  was  serving  about  3G00. 

The  third  question  asked  respondents  to  rate  the  contribution  of  the  service  to 
their  job  performance,  on  a  scale  ranging  from  'nil',  through  'slight',  'useful', 
'very  useful',  to  'essential*.  The  question  was  satisfactorily  productive,  locating 
the  services  in  general  somewheie  between  useful  and  very  useful. 

The  last  question  called  for  an  estimate,  in  cash  terms,  of  the  value  of  the 
service's  contribution  to  the  user's  work  performance,  on  a  six-point  scale  ranging 
from  nil  to  over  £50  a  year.  This  question  worked  very  well  -  better  in  fact  than 
had  been  expected  •  and  allowed  me  to  estimate,  for  example,  that  the  value  of  the 
contribution  of  Daily  Scan  to  each  recipient  was  of  the  order  of  £15.05  a  year. 
This  allowed,  by  scaling  up,  the  production  of  estimates  for  each  bulletin  of  the 

worth  of  the  total  contribution  of  eacti  to  job  performance;  Daily  Scan,  for 
example,  came  out  at  £7525  in  the  best  case,  assuming  respondents  to  be  characteristic 
of  the  population  as  a  whole,  and  £1690  if  the  respondents  were  completely 
unrepresentative.  This  in  turn  allowed  calculation  of  cost-benefit  ratios  by 

comparison  with  the  figures  derived  from  the  study  of  production  cost  in  the  third 
phase  of  the  study,  for  the  Scan  Revi*‘w,  which  cost  £5335  ,  and  produced  best  and 
worst  case  benefits  estimated  at  £0666  and  £2065  respectively,  the  cost-benefit 
ratios  came  out  at  1.58  in  the  best  case  and  0.39  in  the  worst.  (I  should  note 
that  the  costings  had  actually  been  done  shortly  before  the  evaluation  took  place, 
as  part  of  a  general  audit  exercise  within  the  authority,  so  that  there  was  no  need 
to  repeat  them  for  the  study).  Of  course,  in  presenting  these  ratios,  it  was 
necessary  to  provide  an  accompanying  interpretation. 

This  was  an  interesting  study  to  carry  out,  because  it  is  very  rare  to  be  able  to 

produce  more  or  less  genuine  cost-benefit  ratios  at  all.  In  this  case  it  was 
possible  to  produce  ratios  showing  the  relationship  between  production  cost  a'>'l 
perceived  benefit  for  each  service,  with  ’best  case’,  'worst  case'  and  'mid-point' 
estimates  of  each  ratio.  Generally,  the  services  proved  to  be  producing  value  in 
excess  of  their  cost,  except  in  the  case  of  the  Daily  Scan  service,  and  for  that 
service  it  was  possible  to  make  some  suggestions  as  to  how  its  cost-benefit  ratio 
could  be  improved. 

Ironically,  although  both  the  library  authority  and  the  County  Council  were  very 
satisfied  with  the  results  of  the  evaluation,  and  generally  convinced  of  the  value 
of  the  services,  a  further  wave  of  cost-cutting  in  local  authorities,  which  required 
visible  cost  savings  to  be  made,  meant  that  shortly  after  conclusion  of  the  study, 

the  services  were  terminated.  External  circumstances  can  always  force  decisions 
on  administrations  or  service  operators,  and  evaluators  should  always  be  prepared 
to  find  that  decisions  are  occasionally  taken  on  a  basis  of  something  other  than 
pure  reason.  This  is  particularly  so  in  the  area  of  political  decision-making. 


Evaluation  of  an  international  service. 

The  second  study  I  wish  to  describe  is  the  study  of  AGRIS,  carried  out  in  1977  at 

the  request  of  the  Food  and  Agriculture  Organisation  of  the  United  Nations. 

The  resulting  publication  is  the  Report  on  the  Independent  Appraisal  of  AGRIS  (6), 
which  shows  that  you  can  occasionally  win  arguments  with  administrators. 

AGRIS  came  into  being  as  a  result  of  a  recommendation  to  FAO,  in  1970,  by  an 

international  Panel  of  experts,  that  a  single  rapid  current  awareness  service  with 
worldwide  coverage  of  agricultural  information  should  be  created,  with  FAO 
co-ordinating  the  activity.  After  a  series  of  studies*  an  operational  phase  was 
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initiated  in  January  1975,  which  resulted  in  the  production  of  an  agricultural  current 
awareness  service,  producing  monthly  outputs  of  ordered  bibliographical  references 
to  agricultural  documentation,  in  a  printed  version  called  AGRINDCX  and  on  magnetic 
tape.  AGRINOEX  and  the  tapes  were  known  as  AGRIS  Level  1  -  there  was  a  notional 
Level  2,  which  does  not  concern  us  here,  and  for  convenience  I  shall  refer  to  the 
system  as  AGRIS,  collectively.  It  was  modelled  after  the  IMS  system,  the 
International  Nuclear  Information  System  operated  by  the  International  Atomic  Energy 
Agency,  and  the  production  end,  the  AGRIS  Input  Centre,  was  located  in  the  same 
building  as  INIS,  in  Vienna,  using  the  same  machines  and  production  system  as  INIS. 
AGRIS  input  was  submitted  from  designated  national  input  centres,  or  from  regional 
centres  which  coordinated  the  input  from  several  countries.  It  included  both 

conventional  (i  e  journal  and  monograph)  and  non-con vent iona  1  or  grey  literature. 

It  was  decided  at  an  early  stage  that  there  should  be  an  assessment  of  AGRIS  before 
the  end  of  1977  .  It  was  later  agreed  that  the  assessment  should  be  done  by  an 
independent  body,  and  UNESCO  was  requested  to  organise  and  administer  the  study. 
UNESCO  drew  up  the  terms  of  reference,  and  decided  the  composition  of  the  team, 
which  in  the  event  consisted  of  myself  as  Team  Leader,  Professor  Lancaster  from 
the  Graduate  School  of  Library  Science  of  the  University  of  Illinois,  Professor 
Oadran  from  the  Faculty  of  Agriculture  at  Alexandria  University  and  Professor  Haman, 
Head  of  the  Agricultural  Information  Unit  of  the  Polish  Academcy  of  Sciences. 
This  was  an  example  of  what  1  have  mentioned  earlier,  setting  up  a  team  to  represent 
both  developing  and  developed  countries,  socialist  and  capitalist  political  systems, 
information  specialisation  and  disciplinary  knowledge.  If  you  are  lucky,  as  I  was, 
the  team  members  will  reach  a  quick  rapport  with  and  understanding  of  each  other, 
and  will  cooperate  happily. 

The  finally-agreed  terms  of  reference  are  worth  quoting  at  some  length,  so  that 
you  can  see  the  sort  of  thing  you  may  be  in  for  if  you  become  involved  in 
international  studies  of  this  kind.  They  required  consideration  of  the  AGRIS 
programme  in  its  entirety,  and  said  in  part  '..past  achievements  of  AGRIS  shall 
be  evaluated  and  recommendations  for  system  improvement  or  future  action  formulated, 
with  regard  to  four  criteria: 

1.  the  contribution  of  AGRIS  in  assisting  development  of  national  capabilities  for 
information  transfer  (including  institutions,  services  and  personnel)  and  the  role 
of  AGRIS  in  development  of  international  cooperation  involving  Member  States 
and  information  systems; 

2.  the  cost  of  the  AGRIS  programme  in  relation  to  its  benefit  to  Member  States, 
measured  in  part  with  respect  to  criterion  1,  with  particular  emphasis  on  the 
viability  of  the  programme,  future  costa  and  revenues,. 

3.  the  usefulness  of  AGRIS  to  the  various  classes  of  users  of  agricultural  information 
both  in  developed  and  developing  countries; 

A.  the  efficiency  of  technical  aspects  of  information  transfer  within  AGRIS 
(e  g  collection,  processing  and  dissemination  procedures),  of  its  current  and 
potential  products,  as  well  as  training  and  other  ancillary  services,  and  of  its 
central  management  and  the  interface  between  this  and  the  part ic ipst ing  countries. 


These  criteria  were  arranged  in  descending  order  of  importance,  or  to  wut  it 
another  way,  ascending  order  of  the  possibility  of  doing  anything  about  them. 
The  first  criterion  was  clearly  about  contribution  to  broader  policy  goals,  and 
very  much  a  political  objective.  The  second  was  a  cost-benefit  approach,  something 
like  a  user-oriented  study  except  that  because  this  criterion  was  concerned  with 
benefit  to  Member  States  and  not  individuals,  it  also  had  a  strong  political 
flavour.  The  third  was  clearly  user-oriented  in  the  conventional  sense,  and  the 
fourth  criterion  was  management -or iented ,  so  this  particular  study,  like  most 

evaluation  exercises.  had  something  in  it  of  all  the  three  main  types  of 
evaluation  of  which  I  spoke  earlier. 

One  complicating  factor  was  that  AGRIS  had  only  been  in  existence  for  slightly  under 
two  years  at  the  time  the  appraisal  was  begun,  so  that  it  had  had  little  time  in 
which  to  build  up  a  stable  user  community,  particularly  in  competition  with  the 
services  offered  by  the  Commonwealth  Agricultural  Bureaux  and  the  US  National 

Agricultural  Library,  both  of  which  were  long-established,  well-known  and  heavily 
used.  It  had  also  little  time  in  which  to  settle  into  a  steady  state,  with  new 
member  states  constantly  Joining  the  programme,  and  input  procedures  constantly 
changing  and  improving.  Another  cause  of  concern  was  that  the  team  was  to  start 
work  at  the  end  of  October  1976,  and  to  produce  a  report  by  April  1977  ,  which  did 
not  allow  for  a  very  deliberate  approach. 

The  team  first  met  for  two  days  in  Rome  in  October  1976,  during  which  time  we  were 
briefed  on  the  history  and  development  of  the  AGRIS  concept,  and  given  a  large 
collection  of  documentary  material,  including  consultancy  reports,  published  papers 
on  the  subject  and  samples  of  the  products  created.  The  object  was  not  to  have 
a  detailed  picture  of  the  scope  and  operations  of  AGRIS,  but  to  get  a  broad-brush 
portrait  of  the  concept,  sufficient  to  allow  u&  to  begin  to  think.  At  this 

stage,  we  wanted  the  smell  from  the  kitchen  and  a  picture  of  the  dish,  rather  than 
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the  recipe. 

He  felt  we  could  not  establish  a  complete  sampling  frame  for  the  user  population 
worldwide,  because  of  complications  in  the  ways  in  which  AGRINDCX  was  distributed 
in  different  countries,  so  we  decided  at  an  early  stage  to  run  a  tracking  study 
to  find  out  who  the  printed  service  was  i^ctually  reaching.  We  designed  a  simple 

questionnaire,  in  English,  French  and  Spanish,  folded  round  the  front  cover  of 
AGRINOEX  in  a  very  visible  way,  and  addressed  'To  the  person  who  is  going  to  keep 
this  copy  of  AGRINOEX'.  The  questionnaire  asked  for  information  about  the 

distribution  route  and  final  destination  of  the  individual  copy,  who  was  going  to 

use  it,  and  the  purposes  of  use.  We  had  no  great  hopes  of  a  high  return,  but  in 
the  event  we  got  back  ten  per  cent  of  the  questionnaires  in  time  for  them  to  be 
useful  in  writing  the  final  report,  and  over  a  period  of  three  or  four  years  we 

finally  achieved  a  sixteen  per  cent  return.  Given  that  we  did  not  pay  return  postage, 
and  given  all  the  uncertainties  associated  with  this  approach,  this  was  quite  a 
good  response.  It  did  give  us  some  indications  of  wide  penetration  of  the  service, 
and  of  actual  uae  by  a  substantial  number  of  people. 

The  major  part  of  the  study  consisted  of  structured  interviews  in  a  large  sample 
of  countries,  including  all  the  regional  centres  and  all  countries  using  AGRIS  tapes. 
Countries  which  could  not  be  visited  were  surveyed  by  postal  questionnaire. 

Because  we  felt  the  impracticability  of  getting  to  the  ultimate  users,  we  decided 
that  the  target  population  for  interview  should  be  the  information-processing 
cjmmunity,  represented  by  the  AGRIS  input  officers  and  liaison  officers  involved 
at  a  fairly  high  level  in  the  provision  of  agricultural  information  in  their 

respective  countries.  We  chose  the  countries  for  interview  to  be  representative 
of  developed  and  developing  areas,  geographically  representative,  at  different 
levels  of  support  and  input  provision,  and  representative  of  the  world's  political 
groupings.  We  carried  out  some  pilot  interviews,  in  countries  where  team  members 
happened  to  be  and  finally  we  interviewed  in  twenty-five  countries.  The  interview 
schedule  used  corresponded  closely  to  the  questionnaire  distributed  by  post. 

This  questionnaire  was  distributed  by  Professor  Lancaster  and  myself,  as  we  were 
the  team  members  with  most  experience  of  questionnaire  design.  Its  basis  came  from 
a  long  session  at  the  first  meeting  of  the  team,  when  we  broke  down  the  aims  of 
the  study  as  set  out  in  the  terms  of  reference  into  a  number  of  specific  points 
on  which  it  was  possible  and  practicable  to  gather  information.  This  was  a  crucial 
exercise,  the  hinge  on  which  the  whole  study  turned,  and  1  would  like  to  be  able 
to  describe  it  in  detail,  but  it  is  not  easy  to  say  much  about  it.  We  sat  around 
a  blackboard  and  wrote  down  things  like  'What  is  the  national  commitment  in  financial 
resources,  manpower,  equipment,  etc'^'  and  'Has  AGRIS  had  any  effect  on  the 
availability  of  documents?’,  and  developed  question  themes  from  there.  The  essence 
of  the  process  was  that  it  was  a  team  brainstorming  effort.  After  this  exercise, 
designing  the  questionnaire  was  basically  a  technician's  work.  We  had  to  do  it 
in  French  and  Spanish  as  well  as  English,  these  being  the  official  languages  of 
the  system,  and  we  had  to  take  great  care  to  avoid  any  misunderstandings  arising 
from  cultural  differences  among  the  recipients,  any  political  sensitivities,  or 
any  possibility  of  influencing  response  by  apparent  pre- judgment .  It  was  in  the 

end  largely  a  choice-response  questionnaire,  with  encouragement  to  amplify  or  explain 
any  answers.  It  had  thirty-five  questions  altogether,  but  in  spite  of  the  length, 
respondents  did  not  apparently  experience  any  difficulty  in  completing  it. 

After  the  questionnaire  was  designed,  the  team  me  I  aga^i.  It,  to  discuss  it, 

and  to  get  briefing  about  the  countries  and  persons  to  be  visited  for  interviews. 

This  was  a  very  valuable  meeting;  we  argued  at  length  over  the  questionnaire,  revised 
and  reordered  it  and  all  lost  our  tempers  with  it,  and  eventually  reached  the  point 
where  we  shared  a  common  understanding  of  what  the  questions  were  trying  to  find 
out.  This  was  what  the  meeting  was  designed  to  achieve,  and  it  achieved  it. 
When  more  than  one  individual  is  going  to  be  doing  this  sort  of  interview,  a  long 
briefing  and  discussion  stage  is  absolutely  vital. 

Then  we  sent  out  the  questionnaire  by  post,  and  achieved  a  Final  response  rate  of 

twenty-seven  per  cent,  which  is  not  usually  considered  very  good.  However, 

considering  the  inevitable  delays  in  using  the  international  mails,  and  the 
circumstances  that  in  this  case  the  British  Post  Office  sent  some  of  the 
questionnaires  by  surface  mail  rather  than  air  mail,  we  did  not  do  so  badly. 
Where  it  was  practicable  to  do  so,  we  also  sent  interviewees  a  copy  of  the 

questionnaire  several  days  before  their  interview  was  due  to  take  place.  This  is 

generally  desirable,  because,  particularly  when  some  of  the  responses  ere  to  be 
numerical  or  quantitative,  it  gives  the  interviewee  a  time  to  assemble  the  required 
information.  Most  of  the  interviews  took  about  three  hours,  and  most  were  with 
small  groups  rather  than  with  individuals.  In  some  cases,  it  was  possible  to  arrange 
follow-on  interviews  with  others  involved  in  AGRIS  or  in  agricultural  information 
policy  in  the  countries  being  visited. 

During  this  activity,  we  were  also  collecting  information  in  Rome  and  Vienna  on 
the  details  of  the  processing  activity,  since  this  was  within  our  terms  of  reference. 
He  were  given  full  access  to  files,  correspondence  and  reports,  and  in  effect 
collected  every  sort  of  number  we  could  relating  to  the  input,  throughput  and 
outputs  of  the  AGRIS  system. 
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As  soon  as  we  had  reached  the  point  where  the  interviews  had  been  held  and  we  had 
a  number  of  completed  schedules  and  questionnaires  to  hand,  I  put  together  a  first 
draft  of  the  report.  Then  the  team  members  met  for  two  days  in  London  and  we  read 
the  draft,  together  with  other  interview  material  and  each  others'  reports,  and 
then  jointly  roughed  out  a  list  of  the  major  points  which  had  struck  us  during  the 
work  so  far.  This  list  had  about  thirty  items  on  it,  and  we  divided  them  into  two 
groups,  things  we  felt  we  had  discovered  which  we  called  Findings,  and  things  wnicti 
we  felt  should  be  done,  which  we  called  Recommendations.  This  was  done  in  another 
brain-storming  session.  I  then  rewrote  the  report,  prefacing  it  with  the  findings 

and  recommendations,  and  building  in  the  essence  of  our  discussions.  This  document 
was  then  made  the  discussion  paper  for  the  final  meeting  of  the  team,  with  FAQ  and 
UNESCO  representatives,  in  Rome. 

On  the  first  day  in  Rome  the  team  again  went  through  the  draft  report,  tightening 

and  polishing  it  and  making  sure  that  what  we  had  said  was  a  consensus  view. 

It  was  retyped  -  this  was  before  word  processors,  and  how  did  we  ever  manage  without 
them?  -  and  the  next  day  we  recycled  the  whole  thing,  this  time  with  the  FAO 

representatives,  effectively  presenting  the  report  in  the  manner  of  a  consultant 

to  his  clients,  and  accepting  or  rejecting  suggestions  as  they  were  made.  This 
is  always  a  very  important  stage,  because  no  matter  how  professionally  competent 

the  reporting  team,  the  report  must  be  acceptable  to  the  client,  in  the  sense  of 

being  immediately  comprehensible  or  practicable,  if  there  is  to  be  any  hope  of  a 
subsequent  result.  This  is  really  a  negotiation  stage,  in  which  the  report  is  made 
acceptable,  but  without  loss  of  force  or  accuracy. 

To  sum  up  the  AGRIS  experience,  the  stages  were  as  follows: 

1.  A  quick  but  thorough  overview  of  the  problem  context. 

2.  A  brainstorming  session  to  set  the  objectives  of  the  study. 

3.  A  determination  of  the  methods  by  which  information  will  be  sought,  and  the 
sources  to  be  approached. 

A.  Drafting  the  instruments  of  enquiry. 

5.  The  collection  of  information  and  data. 

6.  Preparation  of  a  discussion  draft  report. 

7.  A  Oelphi-like  iterative  process  to  arrive  at  the  draft  final  report,  and 

8.  Presentation  and  negotiation  of  an  agreed  final  report. 

1  have  said  nothing  about  the  findings  of  the  AGRIS  study,  because  the  full  report 
has  been  published  by  UNESCO,  and  I  am  assuming  that  you  will  be  more  interested 
in  the  approach  to  the  problems  of  this  kind  of  assessment  than  in  the  detail  of 
individual  systems.  As  to  the  outcome,  AGRIS  is  still  in  existence,  an  established 
part  of  the  agricultural  information  scene  and  more  effective  than  ever  before, 
90  you  may  conclude  that  the  results  of  this  evaluation  were  positive. 


Evaluating  proposals  for  database  development. 

The  last  exercise  I  want  to  discuss,  rather  more  briefly  than  the  foregoing  studies, 
is  an  activity  which  has  taken  place  six  times  in  the  last  few  years,  and  is  taking 
place  again  in  a  somewhat  different  form.  Strictly  speaking,  it  is  not  concerned 
with  evaluation  of  information  systems  or  centres,  because  it  is  about  the  evaluation 
processes  applied  by  the  Commission  of  the  European  Communities  to  proposals  for 
support  of  database  developments.  The  Leicestershire  study  was  looking  at  something 
that  really  existed  in  a  fully-developed  form.  The  AGRIS  study  was  looking  at  some¬ 
thing  still  in  a  development  stage,  so  had  elements  in  it  of  trying  to  guess  probable 
futures.  The  evaluation  of  the  Calls  for  Proposals  for  the  Commission  is  actually 
almost  entirely  about  evaluation  of  future  possibilities. 

From  time  to  time,  as  a  part  of  its  various  Action  Programmes,  OG  XIII(B)  of  the 
European  Commission  -  the  Directorate-General  which  is  concerned  with  the  information 
industry  and  information  market  -  has  launched  Calls  for  Proposals  for  the  development 
of  advanced  information  services  (meaning,  essentially,  online  or  other  electronically 
-carried  services)  in  various  fields.  Six  such  Calls  have  been  made,  and  more  than 
six  hundred  proposals  were  received  in  response.  In  general,  what  was  offered  to 
successful  applicants  was  25%  of  development  costs,  up  to  a  ceiling  of  200,000  ECU, 
plus  a  possible  additional  25%  to  cover  costs  incurred  by  activities  directed  towards 
increasing  the  European  relevance  of  the  proposed  services. 

Because  of  limitations  on  the  funds  available  for  support  within  these  Calls,  a 
rigorous  selection  procedure  was  necessary,  not  only  to  eliminate  proposals  of  oor 
quality,  but  to  select  the  best  from  amongst  the  good.  Criteria  had  to  be  developed 
to  identify  what  was  meant,  in  Commission  terms,  by  'best',  and  procedures  had  to 
be  developed  which  would  not  only  ensure  that  the  criteria  were  consistently  applied, 
but  which  would  be  demonstrably  free  from  national  or  other  biases  and  fair  to  all 
applicants . 


3-13 


The  criteria  for  selection  were  always  set  out  in  the  text  of  the  various  Calls, 
and  were  decided  by  OG  XIIl(B).  It  was  made  clear  that  support  would  only  be  given 
to  proposals  for  publlcly-available  systems,  and  that  although  associated  printed 
products  would  not  be  discouraged,  the  major  means  of  distribution  had  to  be 
electronic.  Proposals  had  to  be  able  to  demonstrate  that  they  were  unique  in 
character,  either  covering  areas  in  which  there  was  no  provision,  or  covering  areas 
in  a  different  way  and  for  a  different  market  than  existing  provision.  They  were 
required  to  show  that  there  was  an  existing  demand  for  the  proposed  service,  such 
that  the  service,  if  launched,  would  be  likely  to  attract  extensive  use. 
Proposals  also  had  to  satisfy  the  Commission  that  they  were  technically  feasible, 
and  that  the  necessary  input  information  would  be  available  to  the  proposers. 

Proposers  were  also  expected  to  make  a  reasonable  case  that  the  development 
presented  too  high  a  financial  risk  for  them  to  be  able  to  justify  going  ahead  without 
subsidy,  but  that  it  would  ultimately  result  in  a  service  that  would  be  financially 

viable,  without  continued  subsidies.  What  this  meant  was  that  the  proposer  had 

to  have  a  good  commercial  proposition,  but  avoid  giving  the  impression  that  he  was 

asking  for  subsidy  simply  because  the  possibility  of  subsidy  existed.  There  also 

had  to  be  evidence  that  the  remaining  75%  or  50%  of  necessary  funding  would  be 
available. 

Obviously,  because  this  was  a  programme  on  behalf  of  the  European  Communities,  it 
was  necessary  that  proposals  should  be  of  Community-wide  relevance,  and  should, 
in  the  words  of  one  Call,  'have  applicability  for  a  whole  range  of  target  users, 
where  necessary  in  several  languages,  and  that  the  proposed  service  should  be 
marketed  at  EEC  level,  through  cooperation  with  organisations  in  more  than  one 
Member  State  where  appropriate'.  The  Commission  is  trying  to  build  a  common 

internal  market  in  information  services,  by  encouraging  cooperation  between  organisations 
in  the  Member  States,  and  by  breaking  down  language  barriers.  The  criteria  relating  to 
E«iropean  relevance  were  politically  important,  and  were  rigorously  applied. 

There  were  a  number  of  other  policy-related  criteria.  One,  perhaps  the  most 
important,  was  concerned  with  the  extent  to  which  a  proposed  service  contributed 
to  strengthening  or  consolidation  of  European  information  supply.  The  Commission 
wishes  to  develop  a  strong  European  information  industry,  able  to  sell  its  products 
and  compete  successfully  in  world  markets,  and  capable  of  reducing  European 
dependence  on  imported  services.  This  can  be  done  by  strengthening  the  existing 
organisations  which  make  up  the  European  information  industry,  and  assisting  new 
European  organisat ions  to  enter  the  market.  A  service  would  increase  information 
flow  by  bringing  new  user  groups  into  the  marketplace.  A  third  criterion  related 

to  the  ability  of  the  proposed  service  to  improve  the  performance  of  industry, 

commerce  or  scholarship  within  the  Community.  A  fourth  concerned  the  service's 
ability  to  contribute  to  Community  exports,  both  in  itself  as  a  service  which  could 
be  sold  on  world  markets,  and  by  assisting  the  development  of  other  industrial  or 
commercial  products  for  external  sale.  The  last  criterion,  also  of  considerable 
weight,  related  to  the  service's  contribution  towards  diminishing  differences  in 
the  level  of  information  activity  between  Member  States,  which  meant  in  effect  that 
proposals  which  would  be  likely  to  strengthen  the  information  industry  in  a  country 
in  which  that  industry  is  at  present  weak  or  underdeveloped,  or  would  contribute 
to  strengthening  the  information  infrastructure  in  such  a  Member  State,  would  receive 
favourable  consideration,  all  other  criteria  being  satisfied. 

With  such  a  formidable  list  of  criteria,  and  given  the  impracticability  of  constantly 
going  back  to  the  individual  proposers  to  ask  for  additional  information  (although 
we  did  sometimes  seek  clarification  or  more  details),  the  procedure  bad  to  ensure 
that  each  proposal  contained  sufficient  information  to  allow  judgements  to  be  made, 
and  that  the  structure  of  the  information  should  be  as  standardised  as  possible  to 
allow  comparison  between  proposals.  Proposals  were  required  to  be  submitted  on 
a  standard  formsheet,  in  one  of  the  Community  languages  and,  if  possible,  in  English 
also,  if  that  was  not  the  language  of  the  original.  The  formsheet,  which  eventually 
ran  to  24  pages,  asked  for  details  of  the  proposer,  the  service  or  product  to  be 
developed,  the  proposed  marketing  plan,  details  of  the  development  plan  including 
costs  of  activities  and  staff,  source  and  amount  of  existing  funding,  cash  flow 
estimates,  forecasts  of  market  penetration,  and  the  responsi bi 1 i t ies  of  any  partners 
in  the  proposal.  The  formsheet  was  intended  to  be  of  use  to  the  proposer  as  well 
as  the  evaluators,  in  organising  his  thoughts  and  plans  for  the  service  to  be 
developed.  Three  months  ware  allowed  between  the  launch  of  a  Call  and  the  closing 
date  for  proposals. 

When  the  proposals  had  been  received,  each  was  put  through  a  simplified  evaluation 
procedure.  This  required  that  a  simplified  evaluation  formsheet  was  completed  by 
each  of  at  least  two  evaluators,  working  independently.  One  evaluator  would  usually 
be  a  member  of  the  Commission  staff,  and  the  other  would  be  an  independent  expert 
working  under  contract  to  the  Commission.  The  simplified  evaluation  stage  was 
intended  to  eliminate  proposals  which  for  one  reason  or  another  did  not  conform 
to  the  criteria,  or  could  not  be  taken  farther  because  of  lack  of  detail  or  clarity, 
obvious  unfeasibility,  lack  of  European  applicability  or  other  similar  reasons. 

In  the  relatively  rare  case  of  disagreement  between  the  two  simplified  evaluations 
of  a  proposal,  a  third  evaluation  would  be  made  by  another  evaluator,  or  the  conflict 
would  be  resolved  by  discussion  between  the  evaluators.  If  a  proposal  was  rejected 
at  this  stage,  three  reasons  for  rejection  had  to  be  given. 
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Those  which  passed  this  stage  then  went  into  an  extended  evaluation.  Again,  this 
was  based  on  a  standard  formsheet  to  be  completed  by  the  evaluators,  which  was  twenty 
pages  long.  The  first  section  required  the  evaluator  to  spell  out  exactly  what 
was  being  proposed,  in  full  detail.  The  second  section  covered  relationships  with 
existing  services,  including  aspects  of  competition.  The  third  dealt  with  the 
degree  of  contribution  to  Community  objectives,  the  fourth  covered  marketing  aspects, 
the  fifth  covered  the  development  plan  and  the  sixth  dealt  with  the  financial  aspects. 
The  last  section  required  the  evaluator  to  recommend  support  or  rejection,  together 
with  three  good  reasons  to  justify  the  recommendation.  An  experienced  evaluator 
would  take  three  to  four  hours  to  complete  this  formsheet  for  one  proposal. 
Each  proposal,  as  before,  was  reported  on  by  two  independent  evaluators,  but  at 
this  stage  the  evaluators  were  free  to  look  at  all  proposals,  annotate  each  other's 
comments,  add  comments  of  their  own  or  do  another  evaluation,  so  that  many  proposals 
would  carry  comments  from  four  or  more  experts  or  Commission  staff. 

During  the  simplified  and  extended  stages,  each  proposal  was  considered  separately, 
in  isolation  from  the  others.  The  next  step  was  that  all  those  which  had 
successfully  passed  bath  stages  were  discussed  by  the  evaluators,  and  preliminary 
recommendations  made,  to  ensure  that  what  was  being  recommended  would  fit  within 
the  available  budget.  At  this  stage,  proposals  were  dropped,  with  reasons  for  their 
exclusion  recorded.  As  well  as  conforming  to  the  criteria  of  the  Call,  some  regard 
had  to  be  paid  at  this  stage  to  the  balance  of  proposed  funding  between  the  various 
Member  States  and  to  the  possibilities  of  bringing  similar  proposals  together  in 
a  cooperative  effort-  Finally,  for  each  subject  field  or  group  of  proposals, 
evaluation  reports  were  prepared,  summarising  the  main  findings,  giving  the  history 
of  the  evaluation  so  far,  presenting  summary  statistics,  and  listing  all  the  proposals 
submitted.  These  summaries  were  discussed  and  revised,  so  that  in  final  form  they 
represented  the  consensus  view  of  the  evaluator. 

From  this  point,  the  exercise  passed  from  the  evaluators  to  the  administrative 
machinery.  A  committee  of  nominated  expects  from  each  Member  State  would  be  convened, 
to  examine  the  work  of  the  evaluators,  and  to  make  recommendations  for  action  to 
the  next  higher  committee,  which  was  the  Committee  of  Scientific  and  Technical 
Information  (CIDST),  charged  with  the  oversight  of  DC  X1I1(B)'8  action  plan. 
The  nominated  experts  were  in  principal  not  briefed  to  consider  questions  of  policy 
or  promote  national  interests,  but  were  acting  as  technical  experts.  Any  policy 
and  national  inputs  were  the  responsibility  of  CIDST;  CIDST  would,  after 
considering  the  whole  exercise,  formulate  an  Opinion,  in  the  shape  of  a  set  of 
recommendations,  which  would  then  pass  through  a  Commission  selection  committee 
composed  of  representatives  of  interested  Directorates-General .  This  committee 
would  then  make  recommendations  to  the  Director-General  of  DC  XIII,  who  would  take 
the  final  decisions  on  support. 

This  was  a  long-drawn-out  procedure,  but  not,  considering  the  complexity,  excessively 
slow.  It  was  usually  possible  to  move  from  the  closing  date  of  a  Call  to  the 
evaluators'  final  reports  in  less  than  two  months,  and  in  the  case  of  the  last  Call 
we  managed  to  process  over  a  hundred  and  fifty  proposals  in  just  over  a  month. 
It  was  necessarily  a  long  process,  because  at  all  points  it  was  essential  to  ensure 
that  not  only  was  it  as  fair  and  unbiased  as  possible,  it  had  to  be  seen  to  be  fair 
and  unbiased.  Evaluators  from  most  of  the  Member  States  were  used,  selected  by 
the  Commission  and  not  nominated  by  the  Member  States;  Member  States  could  satisfy 
themselves  about  the  procedure  at  both  the  nominated-expe r t s  and  the  CIDST  stages, 
and  CIDST  members  were  free  at  all  times  to  inspect  all  the  documents,  talk  to  the 
evaluators  and  generally  monitor  progress. 

All  this  was  done  in  strict  security  conditions,  in  order  to  protect  the  commercial 
interests  of  the  proposers.  It  is  something  of  a  triumph  to  be  able  to  say  that 
the  integrity  and  validity  of  the  process  have  never  been  disputed. 

Why  I  have  chosen  to  talk  about  this  last  exercise,  whtch  is  'not  about  the 
evaluation  of  existing  centres  or  services,  but  is  really  a  complicated  evaluation 
of  possible  futures,  is  because  it  is  a  good  example  of  a  fairly  rigorous  procedure 
based  almost  entirely  on  informed  subjective  judgements.  All  three  of  the  studies 
I  have  discussed  had,  as  you  can  see,  very  strong  judgmental  aspects,  and  were 
indeed  more  reliant  on  them  than  on  quantitative  measures.  All  three  also  required 
an  awareness  of  political  factors,  which  to  an  extent  dictated  the  approach. 
What  I  have  tried  to  show  is  that  subjective  approaches  and  personal  judgments  can 
be  relied  on,  and  can  produce  successful  outcomes,  if  the  exercise  is  so  structured 
as  tn  minimise  the  risks  of  dependence  on  individual  opinion.  Virtually  no 
evaluation  can  ever  depend  on  objective  data  alone,  and  there  will  always  be  a  need 
to  apply  a  balanced,  disinterested  judgment,  with  due  .'egard  to  the  underlyirg 
policy  implications. 

To  end  on  a  slightly  lighter  note,  if  you  are  sufTering  from  a  lack  of  numerical 
data,  you  have  to  present  your  findings  with  extra  conviction.  The  important 
thing,  at  the  end  of  the  day,  is  to  have  the  client  believe  you. 
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This  paper  uses  the  frasiework  for  evaluation  described  in  *A  Franework  for  Evaluating 
the  Effectiveness  of  lnfor»ation  Centres  and  Services*  for  evaluating  staff  perforaance. 
Inforaation  Centre  staff  are  a  aajor  resource  affecting  the  overall  performance  and 
effectiveness  of  services  provided.  Various  measures  of  staff  performance  are  discussed 
(including  quantities,  qiiallty  and  timeliness  of  staff  outputs),  and  the  relationship 
between  staff  performance  and  the  effectiveness  of  services  (from  a  user  perspective)  is 
demonstrated.  Finally,  alternative  methods  for  measuring  staff  performance  are 
described. 


New  technology,  changer?  in  user  information  seeking  needs,  requirements  and 
behavior,  and  multiple  information  formats  all  have  had  a  profound  effect  on  information 
centre  operations  and  patterns  of  use  that  will,  in  tii»n,  have  an  impact  on  staffing  in 
terms  of  numbers  and  types  of  staff  requjved.  Traditional  methods  of  determining 
information  centre  budgets,  including  personnel  requirements,  have  been  based  on  the 
numbers  of  constituents  served  by  the  centres.  The  nximber  of  professionals  employed  in 
information  centre^  per  10,000  constituents  has  ten»ained  almost  constant  over  the  years. 
Unfortunately,  such  staffing  levels  do  not  reflect  the  realities  of  the  information  age. 

Information  centres  must  provide  access  to  an  accumulating  body  of  recorded 
knowledge.  Since  1970  the  amount  of  knowledge  recorded  throughout  the  history  of 
mankind  up  to  that  time  has  more  than  doubled.  One  of  the  unique  roles  of  information 
centres  is  ho  provide  access  to  all  of  the  published  literature.  Naturally,  they  cannot 
each  hold  all  of  the  literature  ever  published,  but  they  do  provide  access  to  it  through 
other  means.  Thus,  not  only  do  information  centre  staff  have  to  cope  with  an 
ever-increasing  amount  of  new  literature,  they  currently  have  to  provide  access  to  twice 
as  large  a  body  of  literatute  j.n  1988  as  they  did  in  1970. 

Access  to  this  ever-expanding  body  of  literatme  is  p'^ovided  through  an  increasing 
number  of  databases  and  database  services.  These  systems  and  servicof?  are  continually 
being  enhanced  and  upgraded  and  place  an  Increasing  burden  on  Information  centre  staffs, 
in  tenns  of  their  need  to  keep  up-to-date  with  all  changes  so  that  they  can  continue  to 
use  the  systems  and  services  efficiently  and  effectively. 

At  one  time,  libraries  were  concerned  largely  with  books,  then  serials  and 
periodicals?  but  now  they  must  also  store  and  otherwise  provide  access  to  goveinr.ent  and 
technical  reports  and,  because  of  new  technology,  audiovisual  materials,  equipment  and 
software. 

The  recent  growth  in  the  availability  of  information  and  "publications"  in  non-book 
forrmats  raises  several  issues  for  information  centres  and  their  staffs.  In  order  to 
access  information  captured  in  various  non-print  forms  users  need  access  to  the 
equipment  that  is  required  to  "uncapture"  it.  For  example,  a  videocassette  is  not 
readable  without  a  videocassette  recoidp»  and  a  television  monitor?  a  microfiche  cannot 
be  read  directly  without  a  microfiche  reader,  and  so  on.  While  such  equipment  may  be 
available  in  the  home  or  workplace,  equipment  must  also  be  made  available  for  use  in  the 
information  centre  so  that  users  can  review  materials  before  borrowing  them?  and  t.boiie 
v'ho  do  not  otherwise  have  access  to  such  equipment  still  are  able  to  access  the 
information.  This  issue  will  become  more  ciitical  in  the  future  as  increasing  volumes 
of  information  products  are  available  only  jn  nop-prlpt  fo»m. 

An  issue  related  to  the  one  above  is  the  need  for  information  profeps’onal  and  user 
education  and  training  in  the  use  of  the  equipment  needed  to  access  non-print 
information.  Such  education  and  training  will  be  necessary  on  a  continuous  basis, 
particularly  if  use  is  relatively  infrequent.  If  education  and  training  are  not 
provided,  certain  groups  of  potential  user's  will  be  cut  off  from  a.  body  of  ieco.’;ded 
information , 

Information  centre  users  are  becoming  more  information  intensive.  They  are 
increasingly  exposed  to  new  information  technology  —  microcomputers,  videocasyette 
recorders,  etc.  There  has  been  a  great  deal  of  discussion,  in  the  literature,  of  how  as 
the  user  becomes  more  "computer  and  information  literate",  information  centre  staff  will 
no  longer  have  a  role  to  play  as  intermediary.  In  fact,  at  King  Research  we  have 
observed  the  opposite.  Admittedly,  end  users  of  information  do  tend  to  perform  scxne  of 
their  own  information  searches,  particularly  when  they  have  become  comfortable  with 
available  systems.  However,  as  they  do  more  searching  they  recognize  that  more  of  their 
time  is  being  taken  up  with  information  searching  and  retrieval  activities.  They  learn 
that  there  are  numerous  sources  of  information  to  choose  from,  and  that  the  sources 
change  over  time,  in  terms  of  coverage,  procedures  for  use,  etc.  Once  they  recognize 
the  complexity  of  information  searching  and  retrieval,  they  begin  to  return  to  the 
intermediary.  However,  their  experience  has  given  them  a  better  awareness  and 
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understanding  of  infomation  systeiaa^  they  are  more  able  to  communicate  their  needs,  and 
they  are  more  sophisticated  in  terras  of  their  demands  and  expectations. 

One  way  to  discuss  the  complexities  of  information  centre  systems  and  services  and 
their  relationship  to  staffing  patterns  is  through  the  conceptual  framework  discussed 
below. 

Each  function  will  have  a  set  of  services  and  products  associated  with  it.  the 
users  of  which  may  be  information  centre  users  or  staff.  The  functions,  services  and 
products  can  be  divided  into  activities  that  are  necessary  to  perform  and  produce  the 
functions,  services  and  products.  Each  activity  can  be  described  in  terms  of  the 
resources  necessary  to  support  its  performance.  Resources  include  personnel,  equipment, 
information  sources  and  services,  facilities,  supplies,  comnunications.  etc.  Finally, 
each  of  the  resource  conqponents  can  be  described  in  terms  of  its  characteristics. 
Personnel  characteristics  Include  the  type  of  personnel  (professional,  paraprofessional , 
clerical,  etc.),  the  con^etencies  of  personnel  (knowledge,  skills,  attitudes),  the  level 
of  education  of  personnel,  the  experience  of  personnel,  the  salaries  of  personnel,  and 
so  on. 

^  Performance  can  be  measured  at  any  of  the  levels  in  this  framework,  but  the  focus 

of  this  paper  is  performance  at  the  activity  level  and  above.  Performance  of  staff 
Includes  productivity,  quality  and  timeliness  of  performing  these  activities.  The 
effectiveness  of  their  work  is  how  performance  affects  user  satisfaction,  repeated  use 
and  the  purposes  for  which  information  centres  are  used. 

Performance  cannot  be  measured  in  isolation.  A  number  of  factors  influence  the 
performance  of  information  centre  staff.  One  factor  is  the  demand  placed  on  staff  by 
users.  As  the  demand  increases,  so  does  the  workload  of  the  staff  either  directly 
(e.g.,  through  increased  reference  requests)  or  indirectly  (e.g,,  through  increased 
demand  for  more  materials  v/bich  increases  selection,  acquisitions,  indexing,  and  other 
processing  activities;  or  through  increased  administrative  activities).  Thus,  one 
particular  aspect  of  staffing  that  can  be  studied  is  the  relationship  between  current 
and  required  workloads  and  staffing  needs.  Other  factors  that  affect  performance  relate 
to  the  characteristics  of  resources.  For  example,  the  more  competent  a  staff  member  is 
to  perform  a  specific  activity  or  set  of  activities,  the  better  the  performance 
expected.  Similarly,  various  characteristics  of  equipment  used,  information  sources  and 
services,  facilities,  etc.  can  affect  performance  levels  significantly. 

Other  factors  that  affect  staff  performance  (in  addition  to  amount  or  level  of 
input  resources)  Include  management  policies  and  requirements,  organizational  structure, 
attitudes  or  capabilities  of  users,  etc.  Factors  that  influence  effectiveness  (in 
addition  to  output  performance)  include  user  awareness,  attitudes  towards  or  perceptions 
of  a  service,  cYiarges  for  services,  distance  from  service  access  points,  etc. 

Data  collection  for  staff  performance  evaluation  is  done  to  help  ensure  thiat 
sufficient  levels  of  performance  are  achieved.  In  order  to  do  this  we  suggest  that 
several  measures  be  observed  including;  productivity,  timeliness  of  services  and 
quality  of  some  activities  and  services.  There  are  essentially  six  methods  that  can  be 
employed  in  rating  staff  performance  as  follows: 

1)  Observation  of  Productivity.  Productivity  can  be  measured  by  the  number  of 

items  prcxiuced  per  hour.  Output  quantities  may  be  observed  by  logs  O’* 

other  means  over  a  specified  period  of  time  (e.g.,  one  month  or  quarter)  and 
corresponding  input  hours  observed  by  logs,  diaries,  random  observations,  etc. 

2 )  100%  Inspection  or  Observation  of  the  Timeliness  with  Which  Activities  Are 
Performe<r  or  Services  Provid^.  Timeliness  is  determined  by  recording  when 
events  take  place.  Standards  ^or  the  timeliness  of  an  activity  are  based  on  a 
proportion  of  events  that  are  to  be  accomplished  within  a  specified  period  of 
time.  Acceptad>le  performance  might  be  that  no  events  be  accomplished  beyond  a 
specified  time  or  fewer  than  3  percent  accomplished  beyond  that  specified  time. 
All  events  100%  will  be  observed  to  make  sure  that  acceptable  levels  of 
standards  of  timeliness  are  achieved. 

3)  Random  Sampling  of  Output.  In  most  instances  individual  observations  of 
performance  are  difficult  or  expensive.  In  these  instances,  the  output  is 
san^led  and  inspected  for  timeliness  or  quality. 

4)  Random  Spot  Checks.  Spot  checks  ascertain  that  some  activities  are  being 
performed  as  spec  if  led.  For  example,  the  circulation/issue  desk  might  be 
observed  at  random  times  to  insure  that  it  is  manned,  large  queues  have  not 
formed,  and  users  are  being  served  properly,  etc.  A  spot  check  performed  weekly 
or  monthly  can  verify  that  this  is  happening. 

5)  User  Feedback  Including  User  Surveys  Validated  Complaints.  A  survey  of  users 
may  be  performed  in  order  to  ascertain  that  perfonMnee  standards  are  being  met 
(e.g.,  that  a  standards  for  reference  timeliness  and  quality  are  maintained  from 
the  user  perspective) . 

6)  Validated  Patron  Complaints.  Patron  complaints  should  be  encouraged,  received 

and  verified  as  a  means  of  assuring  the  quality  of  work  performed.  All 

complaints  should  be  responded  to  and  the  reasons  for  the  problems  validated  and 
addressed. 
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Each  of  these  methods  is  described  in  more  detail  below. 

Performance  of  a  required  service  is  considered  acceptable  when  the  number  of 
discrepancies  found  during  a  surveillance  period  does  not  exceed  an  unacceptable  number 
of  discrepancies^-  Moat  levels  of  acceptance  found  through  100  percent  inspection  are 
stated  in  terms  of  percentages.  The  percentages  should  be  computed  periodically 
(usually  monthly)  by  dividing  the  number  of  discrepancies  by  a  lot  size  (usually  monthly 
or  quarterly  output).  For  required  services  inspected  by  a  random  sampling,  the  number 
of  faulty  items  or  services  which  determine  a  lot  rejection  is  determined  by  an 
acceptance  level  and  statistical  quality  control  methods  applied. 

In  moat  instances,  the  acceptance  or  rejection  of  performance  will  be  determined  by 
100%  inspection  made  on  a  periodic  basis  (usually  monthly).  In  order  to  facilitate  100% 
inspection  to  observe  timeliness,  it  will  be  necessary  to  maintain  records  (logs)  of 
transactions,  so  that  timeliness  of  events  can  be  determined.  For  example,  a  log  might 
be  maintained  for  each  order,  recording  the  date  of  request,  date  of  order  placement, 
date  of  return,  date  of  delivery,  and  so  on.  The  logs  (maintained  manually  or  on  an 
automated  system)  could  highlight  activities  that  exceed  performance  standards  of 
timeliness.  In  other  instances,  timeliness  of  activities  such  as  delivery  of  periodic 
reports  can  be  observed  through  review  of  each  report  and  its  delivery. 

The  surveillance  records  can  be  maintained  in  any  one  of  several  ways  including 
logs  or  computer  records.  Such  logs  or  records  should  have  some  mecbanism(s)  for 
highlighting  events  that  fall  below  the  acceptance  standards.  Otherwise,  the  logs  or 
records  must  be  reviewed  monthly  to  identify  instances  where  the  standards  are  met  or 
not  met. 

The  second  principal  method  of  inspection  is  random  sampling,  Tn  this  case,  a 
periodic  (usually  quarterly)  lot  of  output  is  sampled  and  inspected.  For  example,  the 
acceptance  level  of  indexing  is  that  no  more  than  five  percent  of  the  items  indexed 
shall  fail  to  conform  to  standards.  A  supervisor  or  someone  designated  by  the 
supervisor  should  determine  whether  sample  outputs  conform  to  information  centre 
standa rds . 

The  person  responsible  for  quality  assurance  should  first  select  the  areas  and 
times  for  random  sampling  using  the  pioceduros  in  the  f^ami-l  ing  guides  and  program  these 
inspections  on  the  schedule.  when  the  seunpling  procedure  does  not  allow  for  specific 
selection  of  inspection  items  during  the  preceeding  quarter,  it  will  be  necessary  to 
show  on  the  schedule  the  date  and  time  the  sample  selection  will  be  accomplished  and 
when  the  inspections  will  be  conducted.  Then  program  into  the  schedule  the  periodic 
checklist  items  to  be  surveilled  during  the  period.  This  schedule  should  show  what  the 
» ef^pons ihl.e  person  is  monitoring  each  day. 

Tally  checklists  should  be  prepared  for  each  sampling  guide  and  will  be  used  to 
record  information  on  observations  and  defects.  Each  observation  in  the  sampling  will 
be  recorded  on  tne  tally  checklists.  These  documents  will  then  become  a  formal  record 
for  later  reference.  The  tally  of  observations  and  defects  at  the  end  of  the  month  will 
be  compared  with  the  acceptable  number  of  defects  appearing  in  the  relevant,  sampling 
guide.  Any  errors  detected  during  the  course  of  the  surveillance,  even  if  not  of 
sufficient  degree  to  render  the  service  unsatisfactory  in  terms  of  the  acceptance  level 
parameters,  will  require  corrective  action  by  the  supervisor  if  possible. 

In  SOTte  instances,  inspection  requires  visual  obsej"vation  of  activities  or  events. 
In  order  to  achieve  such  inspection,  random  spot  checks  of  the  operation  should  be  made 
by  a  supervisor  or  manager.  Such  spot  checks  can  be  made  on  a  weekly  and  monthly  basis 
depending  on  the  observation  wade.  Note  that  the  process  of  spot  checks  will  have  the 
effect  of  the  staff  (or  contractor)  knowing  certain  activities  have  to  be  done  and  on 
time. 
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Relation  of  Sources  of  Competencies,  Competencies,  Services  and 
Products,  Performance  Measures,  Effectiveness  and  Higher  Order  Effects 


Conceptual  Framework  for  Library  Staffing  Studies 
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SEARCHER  COMPETENCIES  AFFECT: 

•  INPUT  COSTS 

•  OUTPUT  PERFORMANCE 

QUANTITIES 

QUALITY 

TIMELINESS 


COMPETENCIES  REQUIRED  OF  INFORMATION  PROFESSIONALS  TO  FACE  THE  CHALLENGE  OF  THE 

INFORMATION  AGE 

A  COMPETENCY  IS  A  GENERIC  KNOWLEDGE,  SKILL  OR  ATTITUDE  OF  A  PERSON  THAT  IS  CASUALLY 
RELATED  TO  EFFECTIVE  BEHAVIOUR  AS  DEMONSTRATED  THROUGH  EXTERNAL  PERFORMANCE 
CRITERIA,  WHERE: 

•  KNOWLEDCT  IS  HAVING  INFORMATION  ABOUT,  KNOWING,  UNDERSTANDING,  BEING 
ACQUAINTED  WITH,  BEING  AWARE  OF,  HAVING  EXPERIENCE  OF,  OR  BEING  FAMILIAR  WITH 
SOMETHING,  SOMEONE,  OR  HOW  TO  DO  SOMETHING 

•  SKILL  IS  THE  ABILITY  TO  USE  ONE’S  KNOWLEDGE  EFFECTIVELY 

•  ATTITUDE  IS  A  MENTAL  OR  EMOTIONAL  APPROACH  TO  SOMETHING,  OR  SOMEONE 


KNOWLEDGE  IS  HAVING  INFORMATION  ABOUT,  KNOWING,  UNDERSTANDING,  BEING  ACQUAINTED 

WITH,  BEING  AWARE  OF,  HAVING  EXPERIENCE  OF,  OR  BEING  FAMILIAR  WITH  SOMETHING,  SOMEONE, 

OR  HOW  TO  DO  SOMETHING. 

•  BASIC  KNOWLEDGE  IN  SUCH  AREAS  AS  LANGUAGE,  COMMUNICATION,  ARITHMETIC 
OPERATIONS,  ETC. 

•  SUBJECT  KNOWLEDGE  OF  PRIMARY  SUBJECT  FIELDS  OF  USERS  SERVED  SUCH  AS  EDUCATION, 
MEDICINE,  CHEMISTRY,  LAW,  ETC. 

•  LIBRARY  AND  INFORMATION  SCIENCE  KNOWLEDGE  SUCH  AS  THE  DEFINITION,  STRUCTURE  AND 
FORMATS  OF  INFORMATION,  ETC. 

•  KNO  WLEDGE  ABOUT  INFORMA  TION  WORK  ENVIRONMENTS  SUCH  AS  THE  INFORMATION 
COMMUNITY,  ITS  PARTICIPANTS  AND  THEIR  SOCIAL,  ECONOMIC  AND  TECHNICAL 
INTERRELATIONSHIPS,  ETC. 

•  KNOWLEDGE  OF  WHA  T  WORK  IS  DONESUCH  AS  THE  ACTIVITIES  REQUIRED  TO  PROVIDE 
INFORMATION  SERVICES  AND  PRODUCE  INFORMATION  PRODUCTS,  ETC. 

•  KNOWLEDGE  OF  HOW  TO  DO  WORK  SUCH  AS  HOW  TO  PERFORM  VARIOUS  ACTIvmES,  APPLY 
TECHNIQUES,  USE  MATERIALS  AND  TECHNOLOGY,  ETC. 

•  KNOWLEDGE  OF  THE  ORGANIZATION  OR  USER  COMMUNITY SERVEDSUCH  AS  THE  MISSION,  GOALS 
AND  OBJECTIVES  OF  THE  USER  OR  THE  ORGANIZATION,  USER’S  INFORMATION  NEEDS  AND 
REQUIREMENTS,  ETC. 


SKILL  IS  THE  ABILITT  TO  USE  ONE’S  KNOWLEDGE  EFFECTIVELY. 

•  BASIC  SKILLS  SUCH  AS  COGNITIVE,  COMMUNICATION,  ANALYTICAL,  ETC. 

•  SKILLS  RELATED  TO  EACH  SPECIFIC  ACTIVTirBElNG  PERFORMED  SUCH  AS  NEGOTIATION  OF 
REFERENCE  QUES’TIONS,  EVALUATION  OF  SEARCH  OUTPUT  S,  ETC. 

•  OTHER  SKILLS  SUCH  AS  MANAGING  TIME  EFFECTIVELY,  BUDGETING  AND  MAKING 
PROJECTIONS,  ETC 


ATTITUDE  IS  A  MENTAL  OR  EMOTIONAL  APPROACH  TO  SOMETHING  OR  SOMEONE. 

•  DISPOSITIONAL  ArnrUDESJOWASD  ONE’S  PROFESSION,  THE  ORGANIZATION  SERVED,  ONES 
WORK  ORGANIZATION,  AND  OTHER  PEOPLE  SUCH  AS  USERS  AND  CO-WORKERS. 

•  PERSONALITY  TRAirS/QUALITIESSVCH  AS  CONFIDENCE,  INQUISITIVENESS.  SENSE  OF  ETHICS, 
FLEXIBILITY,  ETC. 

•  A  TTnVDES  RELA  TED  TO  JOUWORIOORGANIZA  TION  SUCH  AS  WILLINGNESS  TO  ACCEPT 
RESPONSIBILITY,  WILLINGNESS  TO  LEARN,  DESIRE  TO  GROW,  ETC 


AcnvmEs 


REFERENCE  ENTRY  LEVEL 

Solicited  Searches 

1 .  Interview  users  to  evaluate  information  requirements,  and  determine  budgetary  and  time  constraints,  etc. 

2.  Counsel  clients  on  specifying  information  requirements 

3.  Develop  and  implement  strategy  for  obtaining,  evaluating,  packaging  information  and  data  to  meet  users'  needs 

4.  Select  search  methods  (e.g.,  online  and/or  manual) 

5.  Determine  appropriate  source  (e.g.,  databases,  printed  publications,  experts,  etc.) 

6.  Conduct  searches,  using  proper  subject  terms  for  sources  selected 

7.  Review  retrieval  output  for  relevancy 

8.  Provide  information  for  users  in  the  form  of  references  or  source  documents 

9.  Translate  titles  and  abstracts 

1 0.  Refer  users  to  higher-level  reference  staff,  as  appropriate 

1 1 .  Refer  users  to  other  information  services,  as  appropriate 

Unsolicited  Searches 

1 2.  Perform  online  searches  according  to  SDI  profiles 
Other 

1 3.  Answer  ready  reference  questions 

1 4.  Instruct  users  in  reference  methods,  sources,  services  and  policies 

1 5.  Alert  users  to  non-print  and  in-house  developed  sources  of  information 

16.  Perform  bibliographic  verification  as  required 

1 7 .  Recommend  acquisition  of  materials  for  the  reference  and  general  collections 

1 8.  Recommend  materials/data  for  inclusion  in  reference  source  files  (vertical  file;  in-house  indexes) 


REflRENCE 


ENTRY  LEVEL 


SolicUed  Searches 

***knowledge  related  to  literacy,  numeracy,  communications,  etc. 

Subject  knotdedge 

*knowledge  of  the  primary  subject  field  of  users  served  (e.gM  medicine,  chemistry,  law,  etc.) 

*|[nowledge  of  foreign  languages 

Library  AInforrruuion  Science  Knotvledge  (Generic) 

***knowledge  of  definition,  structure,  and  formats  of  information 
’knowledge  of  alternative  approaches  to  the  organization  of  information 
**knowledge  of  alternative  approaches  to  retrieval  of  information 
’knowledge  of  alternative  approaches  to  information  management 
’knowledge  of  available  and  emerging  information  technologies  and  their  applications 
’knowledge  of  completed  and  ongoing  research  in  the  held  and  its  applicability  to  practice 
’knowledge  of  career  opportunities 
’’’knowledge  of  how  to  learn  on  an  ongoing  basis 

Knotvkdge  about  information  work  environments 

’knowledge  of  the  expanding  information  community,  its  participants  and  their  interrelationships  (social,  economic, 
technical,  etc.) 

knowledge  of  the  variety  of  work  settings  and  their  organizational  structures 

’knowledge  of  the  functions  performed  within  the  various  work  settings  and  the  services  and  products  offered 
’’knowledge  of  the  users  of  the  services  and  products,  their  characteristics  and  information  habits 

Knowledge  of  what  work  is  done 

’’’knowledge  of  the  reference  function,  the  range  of  services  and  products  offered  (both  actual  and  potential) 
"’knowledge  of  the  activities  that  are  required  to  offer  the  services  and  produce  the  products 
’"knowledge  of  the  various  resources  that  arc  necessary  to  support  the  activities 
•"knowledge  of  reference/ referral  tools  and  sources  of  information 
•"knowledge  of  reference/ referral  methods  and  techniques 
•^lowledge  of  performance  expected  and  how  it  can  be  measured 

"knowledge  of  job  responsibilities  and  working  conditions  (e  g..  range  of  duties,  probable  compensation,  benefits,  etc.) 
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KNOWLEDGE 


REFERENCE  ENTRY  LEVEL 

Knowledge  of  how  to  do  work 
***luiowledge  of  bow  to  perfonn  the  various  activities 
***l(now)edge  of  how  to  use  the  reference/referral  and  other  sources 
***knowledge  of  how  to  apply  the  reference/referral  methods  and  techniques 
knowledge  of  personnel  procedures 

Knowledge  of  the  o/ganizarion  served  and  its  library  (or  information  center) 

"knowledge  of  the  mission,  goals  and  objectives  of  the  organization  served 

"knowledge  of  the  structure  of  the  organization  and  the  role  of  the  library  (or  information  center)  within  the  organization 
'knowledge  of  the  various  projects  and  key  personnel  within  the  organization 
"knowledge  of  the  policies  and  procedures  relevant  to  the  library  (or  information  center) 

"knowledge  of  the  various  resources  available  within  the  library  (e.g.,  personnel,  equipment,  etc.) 

*"knowledge  of  the  users'  information  needs  and  requirements 
*"knowledge  of  the  collection,  and  of  related  collections 


REFERENCE  MID  LEVEL 

"*greater  depths  of  knowledge  specified  above 

knowledge  of  the  operations  of  other  sections  in  the  library  and  how  they  relate  to 
reference 

*"knowledge  of  available  vendor-supplied  systems,  services  and  products  to  support  reference 
Icnowledge  of  the  contracting  process,  both  in  general  and  within  the  organization 
"knowledge  of  evaluation  methods  and  techniques  to  evaluate  systems,  services  and  products 
"knowledge  of  public  relations  techniques 
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SKILLS 


REFOENCE  ENTRY  LEVEL 

Basic  Skilb 

***Uteiacy,  numeracy,  cognitive,  analytical,  communicatioas,  etc. 

Skilb  Related  to  Spec^  Activities 
Ability  to: 

*perform  each  activity 

**establish  rapport  with  users  and  colleagues 
***coinmunicate  well  by  written,  verbal  and  non-verbal  means 
**conduct  an  interview 

'conduct  meetings  with  individuals  and  groups 
'collect,  analyze  and  interpret  data 

"make  decisions  and  recommendations  based  on  available  information 
'supervise  staff 

'work  independently  and  in  groups 
'develop  criteria  for  evaluation 

Ability  to; 

"make  effective,  timely,  and  well-informed  decisions 

'isolate  and  define  problems  and  develop  the  necessary  criteria  and  action  for  their  solution 
"manage  time  effectively 


REFERENCE  MID  LEVEL 

Skilb  Related  to  Each  Specific  Activity 

*"Skills  listed  above  are  developed  to  a  greater  extent 

Ability  to; 

"perceive  the  needs  of  the  otganization  and  not  just  the  library 
"anticipate  long-range  needs  of  library 

"design  systems  and  procedures  to  improve  library  operations  arbitrate  and  negotiate 
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ATTITUDES 


REFERENCE 

Dispositional  Attiuaies 

Attitudes  Toward  Institutions 
•"Respect  for  profession 
••Respect  for  the  library 
••Respect  for  the  parent  organization 

Attitudes  Toward  Other  People 
Toward  Users 
••Tlespect  users 
••Like  people  in  general 
•••Like  to  help  people 
•Like  to  meet  people 
•Like  to  make  others  feel  comfortable 
•Sensitive  to  others’  needs 

Toward  Others  in  the  Workplace 
"•Respect  co-worker 
•Like  to  work  with  others/as  a  team 
•Like  to  work  on  own 

"Willingness  to  draw  upon  and  share  knowledge  and  experience  with  others 
••Supportive  of  co-workers 
•Enjoy  managing/supervising  others 

Personal  Qualities 
•"Alertness 
•Assertiveness 
•Compassion/Kindness 
"Confidence 
•Cheerfulness 
•"Dependability 
•TJetermination/Tenacity 
••Diplomacy 
••Emotional  stability 
•Fairness 

••Flexibility/Versatility 

•••Imagination 

••Inquisitiveness 
Leadership  ability 
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This  paper  will  discuss  evaluatio}  of  information  centre  resources  (e.g.,  staff, 
equipment,  facilities,  etc,)  In  terms  of  input  costs,  output  performance,  and 
productivity.  It  presents  examples  of  evaluation  of  information  services  (e.g.,  access 
to  technical  reports  and  other  publications,  online  bibliographic  searches,  published 
searches,  etc.)  in  terms  of  their  input  costs  and  output  performance.  The  linking  of 
resource  attributes  or  characteristics  (e.g.«  staff  competencies,  equipment 
capabilities,  facility  layout,  etc.)  to  service  input  costs  and  output  performance 
(e.g.,  quality  and  timeliness  of  services)  is  demonstrated.  Thus,  Centre  managers  can 
make  decisions  about  resources  or  allocation  of  funds  involving  resources  in  terms  of 
services  involved.  A  management  information  system  for  centre  management  is  also 
described. 


In  this  paper,  we  provide  an  example  of  evaluating  information  centre  operations 
and  services.  One  of  the  most  rapidly  growing  technologies  in  the  information  field  has 
been  online  bibliographic  searching.  There  has  been  dramatic  giowth  in  the  numbei  of 
bibliographic  databases  available  online,  the  number  of  professionals  who  use  these 
services,  and  the  average  numbei  of  searches  pei formed  by  or  foi  professionals.  At  one 
time  there  was  broadly  accepted  speculation  that  most  online  searching  would  migrate 
from  being  performed  by  inteiTnediaries  (reference  librarians  or  information  specialists) 
to  being  performed  by  end-users  themselves.  Extensive  efforts  have  been  made  to  make 
searching  more  "uset  friendly"  and  to  tiain  end-users  t.o  perform  their  own  searching. 
In  reality,  even  though  some  searching  is  done  by  end-users,  there  simply  has  been 
little  change  from  having  intermediaries  perform  searches  for  users.  Organizations  that 
have  been  particularly  diligent  in  end-user  training  appear  to  have  merely  created  a 
greater  demand  for  searching;  however,  most  of  this  searching  is  performed  by 
intermediaries  who  support  end-users.  The  principal  reason  that  end-users  (when  trained 
and  when  some  hands-on  experience  is  acquired)  revert  to  support  staff  is  that  such 
searching  is  very  time-consuming,  and  the  time  of  managers,  scientists,  engineers, 
lawyers,  etc.  is  a  scarce  resource.  If  end-users  feel  that  an  inteimediary  can  search 
better  and/or  faster,  professionals  are  likely  to  use  this  resource  to  save  their  own 
time. 


We  estimate  that  professionals  in  the  U.S.  use  reference  and  research  services  an 
average  of  about  3,1  tines  pei  year  (or  an  average  of  9.7  uses  per  professional  who  use 
the  services).  About  17  percent  of  the  transactions  are  considered  to  be  in-depth 
reference  and  research  transactions  (i.e.,  not  quick  look-up).  In  about  79  percent  of 
the  uses,  the  services  were  used  directly  by  the  professionals;  only  21  percent  were 
delegated  to  a  colleague  or  subordinate. 

Users  of  reference  and  research  services  were  asked  how  they  benefitted  by  having  a 
staff  member  from  an  information  centre  perform  the  most  recent  research  foi  them. 
Eighty-three  percent  said  having  an  intermediary  do  the  search  saved  them  (or  their 
staff'  time;  in  fact,  an  average  of  about  7.6  hours  per  search.  About  78  percent  of  the 
end-users  said  that  the  intermediary  could  perform  the  search  faster  and  64  percent  said 
they  could  do  it  better.  This  level  of  confidence  in  delegating  searching  to  the 
intermediaries  is  borne  out  by  the  sati«?faction  with  searches  performed  for  them. 

The  number  of  documents  identified  on  the  most  recent  searches  performed  by 
information  centre  staff  tends  to  be  large,  particularly  when  compared  with  the  number 
of  documents  obtained  and  actually  read  (which  are  indicators  of  relevance) .  Estimated 
average  numbers  of  documents  identified,  c^tained  and  read  are  summarized  below  by  type 
of  document; 


Type  of  Document 

Nuifeer 

Identified 

Number 

Obtained 

Number  Read/ 
Intend  to  Read 

Journal  Articles 

12.4 

6.9 

5.9 

Books 

1.7 

1.0 

0,7 

External  Technical  Reports 

3.8 

1.7 

1.2 

Patent  Documents 

5.4 

2.2 

2.0 

Conference  Proceedings 

2.6 

0.6 

0.5 

Other 

3.1 

1.8 

1.7 

Total 


29.0 


14.2 


12.0 
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Jour^nal  articles  are  the  most  frequently  identified  type  of  document  (43%)  and  the  most 
frequently  read  (49%).  The  average  numbet  of  documents  identified  is  29.0  pei  search. 
About  49  percent  of  these  documents  are  obtained;  and  most  of  the  ones  obtained  are,  in 
fact,  read  (85%),  The  average  number  of  documents  identified  (29.0)  may  seem  high; 
however,  loolced  at  across  all  searches  we  find  that  the  distribution  of  number  of 
documents  identified  is  highly  skewed.  That  is,  a  few  r'.r'a’'ches  account  for  a  large 
number  of  the  documents  identified  (i.e.,  10%  of  the  searches  account  for  about  one-half 
of  the  documents  identified). 

Three  indicators  of  searching  perfo»Tiiance  chat  we  observed  are: 

o  relevance  of  output  to  users*  inforTnation  needs, 
o  numbe»'  of  leferences  in  search  output,  and 

o  time  between  request  and  receipt  of  search  output. 

End-users  were  asked  if,  to  their  knowledge,  there  were  any  relevant  documents  missed  on 
the  most  recent  search  performed  for  them.  For  about  26  percent  of  the  searches,  the 
end-user  did  not  know  whether  relevant  documents  were  missed.  For  the  remaining 
searches,  about  89  percent  bad  no  known  missed  relevant  documents.  The  average  number 
of  missed  relevant  documents  wan  about  foui  per  search.  End-users  said  they  weie 
satisf:od  or  very  satisfied  with  89  percent  of  the  searches  regarding  relevance.  As 
shown  in  Table  1,  the  average  rating  of  satisfaction  was  estimated  to  be  4.15  (on  a 
scale  of  1-5  with  5  representing  "Very  Satisfied").  End-users  were  also  asked  how 
satisfied  they  were  with  the  number  of  references  presented  to  them  in  the  search  output 
(average  being  29.0),  Here  we  find  satisfaction  to  be  a  bit  lower  (average  rating  of 
4.06),  with  84  percent  being  satisfied  or  very  satisfied.  Nearly  all  of  the  end-useis 
who  were  dissatisfied  felt  they  got  too  many  references,  as  opposed  to  too  few, 

TABI.F  1 

PROPORTION  OF  PROFFSSIONAl.S  WITH  VARIOUS  LEVELS  OF  SATISFACTION* 

WITH  ONLINE  BIBLIOGRAPHIC  SEARCHING  DONE  BY 
INFORMATION  CENTRE  STAFF:  1987 


Aspects  of  Online 
Bibliographic  Searching 


Satisfaction  Level 

- - - - - -  Average 

12345  Satisfaction 
(%)  (%)  (%)  (%)  (%)  Level 


Relevance  of  output  to  users’ 
information  needs 

2.2 

0.3 

8.7 

58,0 

30.8 

4.15 

Number  of  references  in 
search  output 

1.9 

2.9 

11.4 

54.7 

29.1 

4.06 

Time  between  request  and 
receipt  of  search  outfrut 

1.7 

0.6 

5.6 

40.6 

51.5 

4.40 

SOURCE:  King  Research,  Inc,  Survey  of  Professionals 


*Satisfaction  Scale:  1 
2 

3 

4 

5 


Very  Dissatisfied 
Dissatisfied 

Neither  Satisfied  NOR  Dissatisfied 

Sat i sf ied 

v'e  r  y  Sa  t  i  s  f  j  ed 


In  terms  of  timeline's  of  response,  we  found  that  the  average  time  to  receive 
search  results  is  3.2  days.  Ab'^'ut  36  percent  of  the  search  results  were  delivered 
within  one  day  and  another  27  percent  on  the  second  day.  About  20  pe/cent  of  i  lie 
searches  took  over  one  week  to  complete.  When  aslced  the  timeframe  within  which  search 
results  were  needed  (to  achieve  the  greatest  value),  mosf.  indicated  times  equal  to  or 
greater  than  the  actual  delivery  time.  The  average  for  the  required  timeframe  is  4.9 
days  (compared  with  3.2  actual  delivery).  Ninety-two  percent  of  the  end-useis  said  they 
were  satisfied  or  very  satisfied  with  the  response  time  of  the  r’^ost  recent  search 
performed  for  them.  The  average  satisfaction  rating  of  4.40  is  higher  than  ratings  fm 
quality  of  search. 

Fnd-users  were  asked  to  indicate  the  importance  of  intennediaries *  knowledge  of  the 
subject  field  searched  and  intermediaries'  skill  in  conducting  the  search.  About  58 
percent  of  the  end-users  consider  knowledge  of  subject  field  to  be  important  or  very 
important.  However,  searching  skill  is  thought  to  be  more  important  as  86  percent 
consider  this  competency  to  be  important  ot  ve:y  ..mportant. 

User  satisfaction  i.s  important  partially  because  means  that  satisfied  users  will 
continue  to  use  inforn'ation  centre  services.  We  have  some  evidence  that  amount  of 
searching  and  user  satisfaction  a^e  correlated.  For  example,  if  we  look  at  satisfaction 
v;lth  relevance  of  output  to  infoi'mation  needs,  the  average  amount  of  searching  by  those 
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end-uaere  who  are  very  satisfied  is  5.12  searches  per  year;  and  for  those  who  are 
satisfied  3.71  searches  per  year.  For  those  who  are  neither  satisfied  nor  dissatisfied, 
the  average  number  is  2.47  searches  per  year;  the  nuitdDer  of  searches  for  those  who  are 
dissatisfied  is  2.00  searches  per  year;  and  for  those  who  are  very  dissatisfied  0.43 
searches  per  year  (see  Figure  1).  Interestingly,  the  number  of  times  the  professionals 
use  alternative  sources  for  their  seaiches  (e.g.,  searching  themselves,  delegating  to  a 
colleague  or  subordinate,  or  using  another  centre)  is  inversely  correlated  to 
satisfaction  with  searches  performed  by  information  centre  staff.  For  example,  those 
very  satisfied  with  information  centre  searches  average  1.5  other  searches;  satisfied  - 
2.3  other  searches;  and  neither  satisfied  nor  dissatisfied  -  4.1  other  searches.  For 
all  throe  levels  of  satisfaction  the  professionals  average  about  four  to  five  searches 
from  all  sources  per  year. 


Level  of  Satisfaction 

Figure  1.  Number  of  Online  Bibliographic  Searches  Done  Per  Year  Through  Library 
Versus  Level  of  Satisfaction  with  Relevance  of  Output  to  Information  Meeds 


Use^  i sfart ’ on  with  number  of  references  presented  in  the  wsarch  output  also  has 
a  similar  rpi  a  i;  ’  onf.h-;p  with  number  of  sea*'ches.  Those  very  satisfied  average  about  5.74 
searches  per  year,  and  those  satisfied  averaged  3.37  searches  per  year.  The 
professionals  who  are  neither  satisfied  nor  dissatisfied  average  about  1.83  searches  per 
year. 


With  t'^melinesG  we  can  compare  satisfaction  versus  actual  response  times.  For 
example,  we  find  that  ave»'age  satisfaction  ratingc.  go  down  appreciably  as  the  difference 
^n  needed  response  times  and  actual  response  times  increases  as  follows: 

n  For  responses  that  are  faste-r  than  needed  time,  the  average  satisfaction  rating 
is  4,45. 

o  When  the  needed  and  actual  response  t’mes  are  the  same,  the  average 

satisfaction  rating  is  4.15. 

o  When  the  actual  response  time  is  gieate'  than  M^e  needed  time,  the  average 

satisfaction  rating  drops  to  3.09. 

Clearly,  response  t'me  ?s  important  when  needed  time  is  considered.  This  is  why 

establishing  needed  time  and  meeting  this  time,  possible,  is  so  important. 

Similarly,  there  is  a  positive  relationship  between  satisfaction  with  search 
response  time  and  amount  of  searches  requested  by  users.  On  the  average,  far  fewer 
searches  are  performed  by  those  dissatisfied  or  very  dissatisfied  with  response  time 
(1.81  and  1.50  searches  per  ynar,  rpspectively)  than  those  who  are  satisfied  or  very 
satisfied  with  response  time  (3,63  and  5.19  searches  per  year,  respectively).  This 
result  would  suggest  that  quality  of  online  searches  Is  also  more  important  than 

timeliness  of  response.  This  assertion  is  reinforced  below  when  we  look  at  the  value 
placed  on  these  two  measures  of  performance. 


Snd-users  who  were  required  to  pay  service  charges  were  asked  how  much  they  paid 
for  their  searches.  The  average  asusunt  paid  was  $260.  For  71  percent  of  Che  searches, 
the  price  was  said  to  be  about  right;  26  percent  said  the  price  was  too  high;  and  two 
percent  said  it  was  much  too  high.  This  is  considered  evidence  of  what  professionals 
are  willing  to  pay  for  their  searches. 

In  order  to  establish  the  relative  value  of  quality  and  timeliness  of  online 
searches,  we  used  a  market  research  tool  called  conjoint  measurement.  This  method 
requires  users  of  online  searching  to  make  judgements  about  a  set  of  alternative 
combinations  of  search  performance  attributes:  quality,  timeliness  and  price.  The 
performance  attribute  of  quality  was  specified  at  three  levels;  high,  medium  and  low 
relevance  of  items  retrieved.  Timeliness  was  also  specified  at  three  levels:  speed  of 
response  within  a  day.  between  one  and  three  days  and  over  three  days.  Price  was 
measured  at  three  levels:  $50,  $150  and  $250.  As  we  indicated  previously,  online 
searching  is  delegated  by  users  to  information  centre  staff  because  users  feel  that  the 
information  centre  can  save  users*  time,  and  the  staff  can  do  the  searches  better  and 
faster. 

The  relative  value  of  quality  and  timeliness  Is  displayed  in  Figure  2. 


RELATIX  TMXflFF  IN  VNLtE  ID  USO  OF  9L\m 
SUtOi  SaEVMCE  MO  lUMNOUO  TIK 


Figure  2.  Relative  Tradeoff  in  Value  to  User  of  Online  Search  Relevance  and 
Turnaround  Time 

If  search  results  are  returned  within  one  day,  but  relevance  of  items  retrieved 
drops  from  high  to  medium,  the  value  decreases  from  $250  to  $150.  Similarly,  if 
revelance  remains  high  but  response  time  drops  from  within  one  day  to  one  to  three  days, 
the  value  drops  from  $250  to  $195.  Thus,  over  all  combinations  of  these  two  performance 
attributes  we  estimate  that  value  of  quality  is  greater  than  value  of  speed  of  delivery, 
although  both  are  appreciable.  Low  relevance  of  items  retrieved  and  response  time  of 
over  three  days  reduces  value  to  only  an  estimated  $10,  Clearly,  both  quality  and 
timeliness  of  response  should  be  kept  at  a  high  level  when  at  all  possible. 


Another  indicator  of  the  usefulness  of  the  online  searches  is  the  purpose  for  which 
searches  are  performed.  EsLiniatcs  of  the  proportion  of  searches  done  for  various 
purposes  are  given  below. 


Pi'opoi'tion 


Purpose  of  Search  (%) 


Research  and  development  29 
Dack^i^ound  research  24 
MLanagcment  or  executive  work  11 
Legal  wozk  9 
Marketing,  sales  or  sales  management  6 
Administration/ finance  6 
Writing  or  giving  presentations  5 
Presentations  3 
Consulting  oz  giving  advice  to  others  3 
Professional  development  2 
Other  2 


Total  100% 


Clearly,  the  searches  are  performed  for  a  wide  variety  of  essential  work. 

Another  way  of  looking  at  the  value  of  searches  performed  by  information  centre 
staff  is  how  much  more  It  would  cost  to  perform  the  searches  if  there  were  •ro 
information  centre.  We  find  that  about  14  percent  of  the  searches  would  not  have  been 
done.  About  one-half  of  the  searches  would  have  been  done*  by  the  profccsionals 
themselves.  Otherwise,  the  searches  would  have  been  delegated  to  someone  else  on  the 
professional's  staff  (12%),  purchased  ftoio  a  broker  (11%)  or  obtained  from  an  external 
source  (13%).  The  cost  of  using  altex-native  means  of  searching  is  estimated  to  be  about 
$145  more  than  is  currently  spent  on  the  searches  (about  $115),  Most  of  this  additional 
cost  is  in  terms  of  the  users*  time  ($105),  but  some  of  it  is  .n  the  additional  cost  to 
purchase  a  search  ($20),  someone  else's  time  ($18)  or  other  expense  ($2), 


SATISFACTION  LEVEL* 


ASPECTS  OF  ONLINE 
BIBLIOGRAPHIC  SEARCHING 

I 

(%) 

2 

(%) 

3 

(%) 

4 

(%) 

5 

w 

AVERAGE 

SATISFACmON 

LEVEL* 

RELEVANCE  OF  OUTPUT  TO 
USERS’  INFORMATION  NEEDS 

2.2 

0.3 

8.7 

58.0 

30.8 

4.15 

NO.  OF  REFERENCES  IN 

SEARCH  OUTPUT 

1.9 

2.9 

11.4 

54.7 

29.1 

4.06 

TIME  BETWEEN  REQUEST  AND 
RECEIPT  OF  SEARCH  OUTPUT 

1.7 

0.6 

5.6 

40.6 

51.5 

4.40 

SOURCE:  KING  RESEARCH,  INC.  SURVEY  OF  PROFESSIONALS 


•SATISFACTION  SCALE:  1  -  VERY  DISSATISFIED 
2 -DISSATISFIED 

3  -  NEITHER  SATISFIED  NOR  DISSATISFIED 

4 - SATISFIED 

5 - VERY  SATISFIED 


Satisfaction  with  Relevance  of  Output 


Relation  of  Search  Perfonnance  as  a  Function 
of  the  Searcher’s  Technical  Skill 


Ranking  erf  Searcher’s  Knowledge  of  Subject  Field 

Relation  of  Search  Performaoce  as  a  Function 
of  the  Seardier’s  Knowledge  of  Subject  Field 


[>ifference  of  Required  Tone  of  Seardi 
and  when  Search  Output  was  Delivered 


Relation  of  Search  Effectiveness  as  a  Function  of  Speed  of  Dchvery 


iML  pr  Satiitactim 


Number  of  Online  Sibliobrrfhic  Semcnes  Done  Per  Year 
Versus  Level  of  Satisfaction  nith  Tine 
Betmeen  Rebuest  and  Receipt  of  Search  Output 


RELATI^  TRNSFF  IN  VAUE  TO  USER  OF  (N.IIC 
SEWCH  REUEVNCE  NO  TURNWOUN)  THE 


RELATIVE  TRADEOFF  IN  VALUE  TO  USER  OF  ONUNE 
SEARCH  RELEVANCE  AND  TURNAROUND  TIME 
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A  PKAMEHORX  FOR  EVALQATIIIG  THE  BFPBCTIVENBSS 
OF  IMFORNATZOa  CEimBS  AM)  SERVICES 


Donald  W»  King 
Joee-Marie  Griffiths 
King  Research.  Inc. 
6000  Executive  Boulevard 
Rockville.  Maryland  20852 
USA 


Evaluation  is  a  necessary  and  integral  part  of  Information  Centre  management. 
Evaluation  must  have  a  purpose  -  it  is  not  an  end  in  itself.  The  need  for  evaluation 
derives  from  the  decision-making  process  which  leads  to  action,  without  the  possibility 
of  alternative  courses  of  action,  one  cannot  conceive  of  a  need  to  evaluate.  Such 
decisions  can  involve  an  entire  organization,  information  services  or  resources 
necessary  to  provide  services  or  produce  information  products.  Evaluation  can  be  an 
integral  part  of  the  decision-making  process  for  designing  or  implementing  a  new  system; 
modifying  an  existing  system;  discontinuing  a  system;  or  continuous  monitoring  of  system 
performance  and  effectiveness. 


Evaluation  is  a  subject  of  considerable  attention  at  the  current  time.  Three 
principal  aspects  of  evaluation  and  information  are  presented  here.  The  first  involves 
the  evaluation  of  information  workers  and  services.  The  second  deals  with  evaluation 
trom  the  perspective  of  users  served,  their  organizations  and  higher  levels  of  aggrega¬ 
tion  such  as  to  industries,  sectors,  the  entire  nation  and  all  nations.  The  third 
aspect  is  how  information  and  information  services  affect  users.  Below,  we  describe 
several  levels  within  information  service  centres.  One  can  perform  evaluation  at  each 
of  these  levels  (i.e..  resource  components,  activi  ti  •=*« ,  services  and  products,  functions 
and  information  units)  .  That  is,  at  each  level  one  can  measure  input  of  resources 
(i.e.,  capital,  labor,  technology,  information,  etc.)  and  output  quantities  produced. 
Such  output  divided  by  input  can  provide  estimates  or  indicators  of  productivity. 

The  output  of  information  services  is  one  of  several  input  resources  to  users.  The 
extent  to  which  this  input  affects  output  and  productivity  is  of  considerable  interest. 
As  shown  in  Figure  1  on  the  next  page,  the  input  resources  (including  information) 
affect  user  activities,  their  functions,  their  organizations,  and  so  on,  up  to  the  point 
that  aggregated  information  use  and  subsequent  productivity  affects  national  goals. 
Below  we  attempt  to  show  how  these  interrelationships  work.  First,  several  definitions 
are  given. 

Definition  of  Terms  Used  in  Productivity 

In  recognition  that  evaluation  terms  have  different  definitions  and  meanings  in 
various  professional  fields  or  specialties,  a  definition  of  terms  is  given  below.  These 
definitions  are  provided  to  insure  a  common  basis  for  discussion  of  productivity. 

Service  -  Services  may  be  found  at  several  levels.  Examples  of  types  of  services 
include  (1)  research  or  consultancy  performed  by  scientists.  (2)  information 
services  such  as  reference  or  access  to  a  collection,  or  (3)  services  provided  by 
administrators.  Such  services  can  be  subdivided  into  discrete  activities  necessary 
to  perform  them  (e.g.,  contacting  a  client,  negotiating  what  is  needed,  performing 
the  research  or  searches,  reporting  results,  etc,).  Each  activity  may  require 
several  resource  components  (e.g..  staff,  equipment,  etc,). 

Input  -  There  are  several  input  resources  necessary  to  achieve  services.  These 
input  resources  include  capital,  staff,  equipment,  instrumentation,  facilities, 
information,  supplies,  administrative  and  support  s;aff,  etc.  Each  resource 
component  can  be  measured  in  several  ways;  for  example,  staff  can  be  measured  in 
Full-Time  Equivalents,  number  of  persons,  hours  of  work,  or  cost  (salary,  fringe 
benefits,  overhead) . 

Output  -  Measures  of  output  services  include  quantities  of  output  (e.g.,  oral  or 
written  reports,  number  of  times  advised,  number  of  searches  performed,  etc.), 
quality  of  service,  timeliness  of  services  provided,  availability  and  accessibility 
of  staff,  etc. 

Productivity  -  Productivity  is  a  measure  of  the  ratio  of  output  divided  by  the 
input.  It  is  formally  defined  as  "a  concept  that  expresses  the  relationship 
between  the  quantity  of  goods  and  services  produced  -  output  -  and  the  quantity  of 
labor,  capital,  land,  energy,  and  other  resources  that  produced  it  -  input".  (U.S. 
Department  of  LaU^or,  Bureau  of  Labor  Statistics,  Productivity  and  the  Economy;  A 
Chartbook  (Bulletin  2172),  Washington,  DC:  Government  Printing  Office,  June  1983.) 
Productivity  is  a  measure  which  links  input  and  output.  The  weakness  of  the 
traditional  measure  of  productivity  is  that  there  is  an  Interdependence  among 
aunount  of  input  resources,  output  quantities  produced,  and  othei  output  measures 
such  as  quality,  timeliness,  etc.  For  example,  a  person's  competency  will  affect 
both  input  (amount  of  time  and  cost  necessary  to  perform  a  seivice)  as  well  as 
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output  (amount,  quality  and  timeliness  of  service).  There  is  usually  an  Inverse 
relationship  loetween  input  (amount  of  services  produced)  and  service  output 
(quality,  timeliness,  etc.). 

Performance  -  Performance  is  an  indication  of  how  well  a  service  or  activity  is 
performed.  It  can  be  measured  in  terms  of  the  output  quantities  produced,  quality, 
timeliness,  availability,  accessibility,  etc.  Other  indications  of  performance  are 
productivity,  efficiency  (i.e.,  how  close  services  or  activities  come  to  achieving 
some  maximum),  etc.  Output  performance  determines  the  effectiveness  of  services 
(see  below).  Measures  of  productivity,  efficiency,  etc.,  are  internal  to  the 
service  and  they  are  used  to  manage  an  operation. 

Effectiveness  -  Effectiveness  is  meacured  from  the  perspective  of  clients  or  users 
of  the  services.  Examples  of  measures  of  effectiveness  include  user  satisfaction 
with  a  service,  repeated  use  of  a  service  and  number  of  timeu  a  service  is  used 
(first  order  effects).  Higher  order  effects  wouDd  include  effect  of  services  on 
the  user's  research  and  consequences  of  the  user's  research,  improved  productivity 
of  the  user's  operation  or  research,  etc.  Presumably,  improved  quality,  timeli¬ 
ness,  availability,  etc.  should  result  in  greater  user  satisfaction,  repeated  use 
and  the  number  of  times  a  service  Is  used.  Thus,  library  service  output  perfor¬ 
mance  partially  determines  the  effectiveness  of  the  service. 

Cost  and  Benefits  -  The  cost  (i.e.,  detriments)  and  benefits  of  a  service  are  the 
unfavorable  and  favorable  comparisons  of  a  service  and  an  alternative  service  in 
terms  of  differences  in  input,  output,  performance  and  effectiveness.  For  example, 
the  value  added  by  the  service  is  that  the  service  costs  less  than  an  alternative 
service,  and/or  the  service  is  better  than  an  alternative  in  terms  of  performance. 
Such  an  alternative  to  a  service  might  be  for  clients  to  perform  the  wor:)c 
themselves  or  to  engage  a  consultant  or  company  to  provide  the  service. 

Indicators  -  Sometimes  it  is  not  possible  to  measure  input,  output,  performance  or 
effectiveness  directly.  Thus,  indicators  must  suffice.  For  example,  and  indicator 
of  the  quality  of  a  research  project  ic  the  number  of  citations  to  an  article 
reporting  the  research.  indicators  of  a  professional’s  competency  are  degrees 
held,  university  attended,  professional  awards  given,  elected  position  in  a 
professional  society,  etc. 

Factors  That  Affect  Service  Input,  Output.  Performance  and  Effectiveness  -  Factors 
that  affect  input  might  include  staff  characteristics,  equipment  attributes,  etc. 
Examples  of  factors  that  might  affect  output  or  performance  (in  addition  to  amount 
of  input  resources)  are  management  (policies,  capabilities,  attitudes,  etc.), 
physical  environment,  attitude  or  capability  of  clients,  etc.  Factors  that  might 
affect  effectiveness  (in  addition  to  output  performance)  are  client  awareness  of, 
attitudes  toward,  or  perception  of  a  service,  a  charge  for  the  service,  distance  to 
the  service,  communication  constraints,  etc.  Some  factors  are  internal  and, 
therefore,  controllable.  Other  factors  are  external  and  less  controllable  by 
management . 

Linlcaqe  of  Service  Incut  to  Output,  and  Performance  to  Effectiveness  -  By  lin)cage 
is  meant  correlation  or  mathematical  models  wliich  show  that  (1)  output  peiformance 
depends  on  input  resources  in  addition  to  other  factors,  (2)  first  order  effects 
(e.g,,  frequency  of  use  of  a  service)  depend  on  output  performance  in  addition  to 
other  factors,  and  (3)  higher  order  effects  depend  on  first  order  effects. 
Examples  of  such  analysis  and  models  include  correlation  and  multiple  regiession, 
conjoint  measurement,  factor  analycis,  multi-aggregate  regression,  etc. 

Linking  Information  Service  Centre  Performance  to  User  Perfoi-mance 

An  example  is  given  below  to  demonstrate  an  approach  to  data  collection  and 
analy.sis  for  information  service  centre  planning  and  evaluation.  The  framework  for  this 
example  is  given  in  Figure  2.  In  the  framework,  one  dimension  involves  three  levels  of 
perspective:  the  information  centre's  service  (e.g.,  online  searching),  user  (e.g.,  a 
scientist  conducting  research),  and  all  of  the  scientist's  organization  (e.g.,  unit). 
The  second  dimension  includes  (1)  input  to  service  or  activity,  (2)  the  actual  service 
or  activity  and  its  resources,  and  (3)  output  of  service  or  activity.  In  the  figure, 
arrows  signify  relevant  quantitative  relationships.  For  the  information  centre's  ser¬ 
vice  perspective  example,  the  service  is  online  searching  and  resources  are  searchers, 
terminals,  search  tools,  support  staff  (e.g,,  to  do  photocopying,  typing,  etc,), 
communication,  search  systems,  pliotocopying,  etc.  Associated  with  the  resources  are 
costs  which  are  dependent  on  attributes  of  the  resources  (arrow  (a))  and  output  number, 
quality,  timeliness,  etc.  of  searches  (arrow  (c)).  Likewise,  output  performance  (in 
terms  of  nunOaer  of  searches  conducted,  quality  and  timeliness)  depends  partially  on  the 
attributes  of  the  resources  (arrow  (b) ) ,  The  relationships  of  resource  attributes  to 
input  cost  and  output  performance  are  quite  clear.  For  example,  better  searcher 
competencies  (i.e.,  knowledge,  skills  and  attitudes)  usually  cost  more  (in  salaries)  but 
should  also  yield  more  and/or  better  search  output.  Performance  of  online  searching  can 
also  be  measured  by  cost  per  search  or  productivity  (searches  per  hour  or  cost).  These 
relationships  are  also  designated  by  arrow  (c> .  Regarding  productivity,  we  feel  that  it 
is  best  to  incorporate  quality  and  timeliness  with  such  comparisons,  such  as  by 
measuring  cost  per  search  at  levels  of  quality  or  at  least  above  an  acceptable  level  of 
quality  and  timeliness. 
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SERVICE  OR 
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Similarly,  research  conducted  by  a  scientist  is  given  as  an  example  of  the  user 
perspective.  To  perform  this  activity,  in  addition  to  scientist's  time,  he  must  have 
such  resources  as  equipment,  instrxunentation,  facilities,  support  staff  and  information. 
The  amount  and  quality  of  information  used  by  the  scientist  depends  partially  on  the 
output  performance  of  the  online  search.  This  relationship  is  designated  by  arrow  (f). 
In  turn,  the  amount,  quality  and  timeliness  of  information  will  affect  cost  of  perform¬ 
ing  the  activity  (arrow  (g))  and  output  amount,  quality  and  timeliness  of  research 
(arrow  (h) ) .  Productivity  of  the  scientist  is  shown  by  arrow  (i).  Thus,  the  output  of 
the  information  service  should  affect  the  productivity  of  scientists.  In  turn,  output 
performance  of  scientists'  activities  should  affect  the  input  cost  and  output  perfor¬ 
mance  of  their  units,  and  hence,  all  of  the  organization  (arrows  (k) ,  (1),  and  (m) ) . 

Measures  and  Models 


Measures 


For  each  information  service  there  are  subactivities  and  resources  that  are  used  in 
the  cervices  (A  in  Figure  2).  In  addition  to  the  numbers  or  amounts  of  the  resources, 
one  can  characterize  them  concerning  attributes  that  are  likely  to  affect  input  cost  (B) 
and  output  performance  (C)  .  Such  attributes  would  include  staff  ccmipetencies  (know¬ 
ledge,  skills,  attitudes)  or  indicators  of  competencies  such  as  educational  background, 
training  and  years  of  experience?  equipment  capabilities  such  as  baud  rates,  storage 
capacities,  down-time,  etc.?  photocopy  or  reproduction  quality?  microfilm  viewer  reada¬ 
bility?  and  so  on.  Cost  data  can  be  subdivided  into  fixed  one-time,  fixed  recurring  and 
variable  elements  and  by  direct  and  indirect  costs.  Output  performance  (C)  can  be  iden¬ 
tified  and  measured  in  terms  of  quantities  produced,  quality  (probably  more  than  one 
measure  per  service),  timeliness  (probably  both  by  response  times  and  difference  in 
negotiated  time  and  actual  time)  and  any  other  attributes  that  might  appreciably  affect 
user  satisfaction  and  continued  use  of  the  service.  We  have  found  that  it  is  sometimes 
useful  to  set  an  acceptable  level  of  performance  quality  and  timeliness  so  that  one  can 
estimate  the  frequency  of  unacceptable  performance  and,  perhaps,  later  implement  a 
quality  control  program  for  some  services. 

In  particular,  one  should  identify  the  activities  performed  by  users  and  determine 
the  extent  (i.e.,  hours)  to  which  they  are  performed  by  users  (D)  .  For  each  activity, 
the  resources  used  are  determined,  particularly  regarding  types  of  information  provided 
by  the  information  services.  We  also  obtain  attitudes  concerning  the  relative  impor¬ 
tance  of  information  compared  to  other  resources  in  performing  the  various  user 
activities.  We  measure  the  cost  (E)  to  users  of  identifying,  acquiring  and  reading 
information.  These  costs  include  user  labor,  support  labor,  equipment,  purchases  of 
materials,  and  so  on.  We  also  measure  indicators  of  user  output  associated  with  each 
generic  activity  (F) ,  Such  output  would  include  number  of  proposals  or  plans  written, 
research  reports  (e.g,,  laboratory  notebooks)  prepaied,  journal  articles  or  other 
publications  written,  consultations  done,  sales  calls  made,  and  so  on.  In  addition,  for 
critical  incidents  we  estimate  the  contribution  that  readings  (of  books,  journal 
articles,  technical  reports,  etc.)  have  on  saving  labor  and  equipment,  improving  quality 
or  speeding  up  completion  of  an  activity,  and  so  on.  Finally,  we  measure,  to  the  degree 
possible,  how  user  performance,  in  turn,  affects  the  unit's  or  the  organization's  total 
input  costs  and  output  performance. 

Evaluation  Models 

We  have  established  correlations  and  other  quantifiable  relationships  between 
service  input  costs  and  output  performance?  service  output  performance  and  user  input 
cost  or  output  performance?  user  input  cost  and  user  output  performance?  and  user  output 
performance  and  unit/company  input  costs.  These  relationships  or  links  are  developed 
largely  by  multiple  regression  and  conjoint  measurement  models.  Such  models  can  be 
employed  because  our  surveys  yield  several  hundred  observations  of  critical  incidence. 
The  conjoint  measurement  permits  us  to  determine  the  relative  contribution  that  service 
performance  attributes  make  over  various  levels  of  the  attributes.  For  example,  one  can 
determine  how  much  use  would  be  lost  over  various  combinations  of  search  quality  and 
speed  of  response. 

Each  arrow  (a,  b,  j)  in  Figure  2  represents  relationships  that  will  be 

modelled.  Examples  of  models  for  each  arrow  are  as  follows: 

(a)  Cost  as  a  function  of  searcher  competencies 

(b)  Quality  of  search  as  a  function  of  searcher  competencies 

(c)  Cost  per  search  or  productivity  (at  acceptable  levels  and  unacceptable  levels 
of  quality/ timeliness) 

(d)  and  (j)  Contribution  of  costs  to  total  organization's  costs 

(e)  Efficiency  of  resources  and  service 

(f)  Amount  of  information  input  as  a  function  of  search  quality,  timeliness, 
accessibility 


(g)  User  input  cost  as  a  function  of  information  input 


(h)  user  output  perfomance  as  a  function  of  information  input 

(i)  User  productivity  for  various  activities  as  a  function  of  information  input 

(k)  and  (n)  Unit  input  costs  as  a  function  of  user  output  performance 

(l)  Unit  output  performance  as  a  function  of  user  output  performance 

The  examples  above  reflect  only  dependent  and  principal  independent  variables.  Other 
independent  variables  are  introduced  to  account  for  other  sources  of  variation.  Our 
experience  has  shown  that  the  measurable  relationships  become  weaker  as  the  ability  to 
observe  or  measure  becomes  more  difficult.  Nevertheless,  we  have  found  positive 
correlations  in  most  such  instances. 

Cost  and  benefit  analysis  can  be  done  by  comparing  each  information  service  to  its 
least  expensive  alternative  (or  alternative  that  is  likely  to  be  employed  by  users). 
Then,  by  means  of  the  models,  one  can  compare  the  current  service  wwith  the  alternative 
concerning:  service  input  costs,  service  output  performance,  user  input  costs,  user 

output  performance,  and  so  on.  From  the  perspective  of  the  entire  organization,  if  the 
comparison  is  favorable  (e.g.,  costa  less),  the  difference  should  be  recorded  in  a 
benefits  column,  and  if  It  Is  unfavorable  (e.g.,  less  output),  the  difference  should  be 
recorded  in  a  cost  (i.e.,  detriments)  column.  It  is  emphasized  that  the  entries  of  cost 
and  benefit  should  be  in  terms  of  dollar  expenditures,  other  measures  (e.g.,  number  of 
searches,  quality  of  searches,  etc.),  or  merely  statements  of  advantages  or 
disadvantages . 

Finally,  regression  and  correlation  have  been  used  by  Hayes,  Terleckyj,  Baumol, 
Braunstein,  Jonsher  and  others  to  demonstrate  that  purchase  and/or  use  of  information 
and  information  equipment  or  services  is  correlated  to  an  organization's  or  industry's 
productivity.  Griliches  has  also  demonstrated  the  linkage  of  user  productivity  (human 
capital)  to  the  nation's  productivity. 


IfOlCATORS  OF  EFFECTS  OF  INFORMATION 


•  Importance  of  information  as  an  input  resource 

•  Salaries  positively  correlated  with  amount  of  reading 

f  Productivity  positively  correlated  with  amount  of  reading 

•  Other  indicators: 

savings  in  time  and  equipment 
-  activities  completed  faster 

ACTIVITIES  PERFORMED  WITH  GREATER  QUALITY 


NEW  ACTIVITIES  INITIATED 
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t  Salaries  and  scientists  and  engineers  are  positively 

CORRELATED  TO  AMOUNT  OF  READING  DONE 


OnCR  IIOICATORS  OF  T)C  EFFECTS  OF  IITORMATION 


•  SAVINGS  IN  Tire  AND  EOUIPreNT 


Journals 

Books 


X  OF  RFAniMKS 

25X 

39X 

7ZX 


AVS.  SAI/IMCS 

J385 

<280 

<706 


Technical  Reports 


6-8 


EXMFUES  OF  CONSEOJENTIAL  VN.UE 


I  A  NUCLEAR  SCIENTIST  INOICATEO  THAT  A  RECENT  READINS  OF  A 
TECHNICAL  REPORT  ON  ELECTRICAL  DISCHARGES  IN  A  VACUUM  "SAVED 
TIME  BT  LETTING  ME  PERFORM  WORK  THAT  COULD  NOT  HAVE  BEEN  DONE 
OTHERWISE. ■ 


t  A  SCIENTIST  INOICATEO  THAT  HIS  READING  OF  A  COAL  GASIFICATION 
PROCESSING  REPORT  “PROVIDED  COLLATED  INFORMATION  IN  A  SINGLE 
volume,  which  mean  that  he  DIO  not  have  to  do  THE 

COLLATION." 


•  A  BASIC  ENERGY  SCIENTIST  SAID  THAT  HIS  READING  OF  A  REPORT  ON 
DOUBLE  EFFECT  ABSORPTION  COOLING  "PREVENTED  SAVINGS  TO  DO  A 
COMPLETE  DESIGN  OF  A  SYSTEM  INCLUDING  DESIGN  CALCULATIONS." 


CmeR  IJCICATORS  OF  T*  EFFECTS  OF  INFOlWATION 


•  ACTIVITIES  PERFORltD  WITH  CREATES  OJALm 


X  OF  READINGS  PROP.  INCREASE 
JOURNALS  31X  160X 

Books  m  55X 

OTFCR  INDICATORS  OF  TWE  EFFECTS  OF  IlfORMATION 


•  ACTIVITIES  COMPLETED  FASTER 


X  OF  READINGS 
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Books  iax 


OTTCR  IWICATORS  OF  UC  EFFECTS  OF  INFORMATION 


•  NEW  activities  INITIATED 
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Books 
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USER  PMXUCTIVirr  iS  C0RRELA1ED  TO  MDUNT  OF  READING 


AMOUNT  OF  READING  IS  POSITIVELY  CORRELATED  TO  PRODUCTIVITY 
MEASURED  IN  TERMS  OF: 


•  NUMBER  OF  FORMAL  RECORDS  OF  RESEARCH  WRITTEN 


•  NUMBER  OF  FORMAL  PUBLICATIONS  WRITTEN 


•  NUMBER  OF  PROPOSALS  OR  RESEARCH  PLANS  PREPARED 


(  NUMBER  OF  FORMAL  ORAL  PRESENTATIONS  MADE 


•  NUMBER  OF  TIMES  IN  WHICH  SCIENTISTS  ARE  CONSULTED  OR  SIVE 
SUBSTANTIVE  ADVICE 

ILOSSARY  OF  COST  FINDING  TERMS 


•  COST.  The  AMOUNT  OF  MONEY  OR  OTHER  CONSIDERATION  EXCHANGED 
IN  ORDER  TO  PROVIDE  A  SERVICE.  ACTIVITY  OR  FUNCTION. 


t  COST  FINDING.  A  less  FORMAL  METHOD  (THAN  COST  ACCOUNTING)  OF 
COST  DETERMINATION  USING  AVAILABLE  FINANCIAL  DATA  AND 
RECASTING  AND  ADJUSTING  IT  TO  DERIVE  THE  COST  DATA  NEEDED. 


•  EXPENDITURE.  The  COST  of  acouiring  goods  and  services. 


•  EXPENSE.  The  use  of  financial  resources  for  providing  goods 
AND  services  regardless  of  WHEN  EXPENDITURES  ARE  MADE. 


•  DIRECT  COSTS.  COSTS  THAT  ARE  READILY  ATTRIBUTABLE  TO  A 
SPECIFIC  SERVICE.  RESOURCE.  OR  ACTIVITY. 


t  INDIRECT  COSTS.  COSTS  THAT  ARE  NOT  EASILY  ASSIGNABLE  OR 
READILY  ATTRIBUTABLE  TO  ANY  ONE  SERVICE.  ACTIVITY.  OR 
FUNCTION. 

Indirect  Operating  Costs.  These  costs  include  centrally 

BUDGETED  ITEMS  (E.G..  UTILITIES.  RENT.  INSURANCE.  ETC.) 
THAT  ARE  NECESSARY  TO  THE  GENERAL  OPERATION  AND 
MAINTENANCE  OF  THE  LIBRARY  SYSTEM. 

Indirect  Support  Costs.  Costs  for  support  services  that 
BENEFIT  overall  ADMINISTRATION  OF  THE  LIBRARY  SYSTEM  AND 
ITS  SERVICES  (E.G..  ADMINISTRATION.  ACCOUNTING. 
PERSONNEL.  ETC.). 
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a.oss/tfnr  or  cost  finding  terms  (continued) 


•  FIXED  COSTS.  Costs  that  remain  constant  regardless  of 
CHANGES  IN  the  ACTIVITY  LEVEL  OR  METHOD  OF  PROVIDING  THE 
activity. 


•  SEMI -FIXED  COSTS  (’STEP  COSTS").  COSTS  THAT  ARE  FIXED  AT  A 
CERTAIN  LEVEL  OF  ACTIVITY,  BUT  CHANGE  WITH  LARGE  INCREASES  OR 
DECREASES  IN  ACTIVITY  LEVEL.* 


•  VARIABLE  COSTS.  COSTS  that  change  Directly  with  changes  in 
THE  LEVEL  OF  ACTIVITY  OR  METHOD  OF  PERFORMING  THE  ACTIVITY* 

Labor  Salaries,  includes  only  labor  time  associated  with 

PROVIDING  a  service  OR  PRODUCING  A  PRODUCT.  (DOES  NOT 
INCLUDE  FRINGE  BENEFITS.) 

Other  Variable,  includes  all  other  costs  associated  with 
PROVIDING  A  service  OR  PRODUCING  A  PRODUCT. 


•  INCREMENTAL  COSTS.  ThE  amount  of  VARIABLE  COSTS  associated 
with  each  transaction  or  unit. 


•  AVERAGE  OR  LINIT  COST.  The  financial  RESOURCES  NECESSARY  TO 
PROVIDE  ONE  UNIT  OF  SERVICE. 


•  DEPRECIATION.  A  method  OF  ALLOCATING  THE  EXPENSE  OF  AN  ASSET 
TO  THE  PARTICULAR  TIME  PERIOD  FOR  WHICH  COST  ANALYSIS  IS 
PERFORMED. 


•  ALLOCATED  COSTS.  The  division  of  costs  among  services.  For 

EXAMPLE,  THE  DIVISION  OF  TERMINAL  OR  COMPUTER  COSTS  AMONG 
CATALOGING,  INTERLIBRARY  BORROWING,  AND  ONLINE  BIBLIOGRAPHIC 
SEARCHING. 


•  FULLY  ALLOCATED  ACTIVITY  COST.  The  total  COST  of  an 
ACTIVITY,  including  BOTH  DIRECT  AND  INDIRECT  COSTS. 


•  ECONOMIES  OF  SCIALE.  Situation  in  which  average  (or  unit) 

COSTS  DECREASE  FOR  INCREASING  NUMBER  OF  TRANSACTIONS  OR  UNITS 
OF  SERVICES. 
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EOMMIES  OF  SCALE 

•  ECONOMIES  OF  SCALE  DUE  TO  SPREADING  OUT  FIXED  COSTS 

t  OTHER  SOURCES  OF  ECONOMIES  OF  SCALE 

Batching  Services  for  Operational  Economies 
-  Optimum  Allocation  of  Staff 

•  PERFORMANCE  AS  A  FUNCTION  OF  QUANTITIES  OF  SERVICE 

S^AFF 

Technology 

(  DISECONOMIES  OF  SCALE 
Administration 
Communication 


>1 
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TAMXI 

KESOUICC  KIFOKMANCC  AND  EFTECnVEWSS  INMCATOaS 


RESOURCE 

INPUT 

OUTPUT 

0PERAT10NA1  PERFORMANCE 

EFTTiCnVENESS 

OTHER  DERTVEO  INDICATORS 

COLLECTION 

Co«l<S) 

Oty- (Tiilcs.  l(enH) 

Cc»t/Oty 

Extent  of  Uw: 

Cost  per  capita 

Routtrcck 

Anrrtwies 

CircsilaiKMi 

Annbutei 

AddllMAS.  «ithJniMls 

In-hosisc  ipatfOA.  staff) 

Avg  coKiTevd  of  saturfacoon 

AdctiticMU.  wichjrawah 

Quality  (Scope  -  whai 

OiyyCoai 

ILL 

covered,  camprehensiveness  — 

Purpose  of  use 

Cosi/exient  of  use 

ho*  well  covered) 

Usif  vaiiafacbon  with 

EsicflitrfuiC’  capita 

AvailatMbry  (ttoun  available. 

Avg^  coM/level  of  qual 

quabty.  availability. 

slock  unavaiUMe  —  m 
proccu.  (M  loan, 

actetiibibiy 

Turnover  rate* 

cte) 

Avg  cocOevel  of 

Fill  rates  (eg.  nilc. 
authot*.  subject,  browsing 

Avg  uie'level  of  satisfaction 

AccessttRhiy  (open  w 
dosed  stacks.  ms-shcKed. 

avaiWuliiy 

etc) 

Oiy  capsia 

eic) 

Multiple  use 

ILfi'quantities 

Avg.  csM  level  of 
accessibility 

Multiple  users 

ILB'Cstenl  of  use 

Propoftiem  of  cnIIcciion  on  loan 

STAFF 

C<ist  (S) 

Qtv  (head  count.  FTF.s. 

Com  Qty 

Duipul  Ilf  work 

C'om/uiiii  of  output 

Resoufcn 

htwrsnfwork) 

Atlnbuiei 

Attnbuies 

Qty.  Cost 

Fringe 

Quality  (produetiviiy. 

Avg  cwsl'levd  of  qua) 

Management  or  usci  satisfacimn 

Com  capita 

accuracy  of  work,  etc  ) 

Avg  cost/fevd  of 

with  quality 

Availabibty  (actuid  hours 
worked) 

avaMabiliiy 

Avg  cost  1cvH  of  satisfaciHin 

EQUIPMENT* 

C0!ll(S) 

Qty.  (AfflI.  of  equipment. 

CoM'Qiy 

No  of  uses 

Cost  capita 

SYSTEMS 

Reiiourcrt 

no.  flfhoun) 

Aitrihuicf 

Acmbuie* 

Qty /Cost 

No  of  u«rs 

Cost  uw 

Quality  (rdiabiiity. 

Avg  coai4cvel  of  qual. 

Purpoaeof  use 

Uses  ctpiia 

screen  roadability,  etc.) 

Avg  COM /level  of 

User  sarisfacbun  with 

Hours  of  use/capiu 

Timeluaess  (response  time. 

tuneiiness 

leaiwres.  quafaiy.  iimelii>es.s. 

baud  rate,  etc.) 

Avg  cost/)es«)  of 

accessibiNiy 

Proportion  of  lotal/available  hours 
actually  used 

Aocessibiliiy  (no.  of 
hours  available) 

accessibitiiy 

Avg  cn*t/1evel  of  satisfaction 

FACILfTTES 

COM(S) 

Qty.  (Total  space  m  sq.  ft . 

Cott'Qiy. 

No  erf  visits 

Cmt/viMi 

Resource* 

laiaMe  space) 

Anributes 

Aniburcs 

OiyvCoM 

No  erf  viMlor 

Cosi'capiiB 

^lality  (ease  of  use. 
pfeasanmeas  of 

Cost /hours  (rf  use 

surraundinfs) 

Avg  cost  Aevel  of  qual 

Hours  of  use 

Visils'capita 

Availability  (hour*  open) 

Avgcostylevelof 

^rpeaeofuse 

Vsitors'capiu 

Accessibility  (avg. 
distance  in  time  or  miles 

avaitebitity 

Hours  of  use/capita 

lo  constitiients  served. 

User  satisfaction  with 

Avg.  cQsi/lcvei  of  satisfaction 

amt  of  parking,  entrance 

Avg  coai'levci  of 

quality,  availabihtv.  accessibiliiy 

for  disabled,  etc  ) 

accessibiiity 
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30,000 

Annual  Amount  of  Savinos 


Proportion  of  Journal  Readings  in  Which  Various  Levels  of  Savings 
are  Achieved 
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SiuuMry 

A  busineesHineiiied  apivoach  lo  mformifinn  cma  pUiming  end  evaluatioo  is  preeenaed.  Hie  need  far  ouipitt  mensires 
and  quaniiaiive  expression  of  service  value  is  described  and  a  meihod  (rf  i^anning  inconmating  these  coocqxs  is 
provided.  Both  tofHfawn  and  bottom-up  planning  res|)OQSibilities  are  tdeati^  and  the  specific  steps  invtrived  in  a 
business  planning  process  «e  given.  The  competitive  nature  of  information  service  operation  is  emphasized  dirou^iouL 


Introdaetha 

All  information  service  organizations  operate  in  a  competitive  arena  -  be  ihey  Ubrarks  serving  the  pubic  sector  or 
specialized  information  centers  in  govemmem  or  industrial  RAD  environments.  Hiey  all  compete  with  odier 
organizations  for  finite  and  scarce  resources  in  a  zero-sum  game.  There  are  winners  and  losers,  onfonnnaiely,  in  the 
competitive  arena  and  the  losers  are  not  necessarily  the  organizations  that  have  the  least  to  offer.  They  are  <rflcn  the 
(vganization  least  prepaiod  to  express  their  vahie  and  contribution  in  terms  undersmod  by  the  funders. 

In  business,  govemmem  and  educational  instfaitioos,  the  mavcrsal  terms  understood  all  decision  maken  are  cost  and 
benefits  as  measured  in  quantitative  tenns.  Lord  Kelvin  said,  ”...  that  when  you  can  fneasiae  what  yon  are  q^eaking 
about,  and  express  it  in  numbers,  you  know  something  about  it;  but  when  you  cannot  express  it  in  numben,  your 
Knowledge  is  of  a  meagre  and  unsi^actory  kind  The  modem  executive  says,  on  a  similar  note,  "show  me  the 
numbers." 

When  the  U.S.  government  decided  to  pull  the  plug  on  information  analysis  centers  in  the  1970*8.  the  decision  was  ckariy 
driven  by  economics.  Labor  costs  were  rising  and  the  return  on  invesunem  the  traditional,  labor-intensive  qtproadh 
was  not  explicitly  defined.  The  move  to  on-hne  computer  search  systems  was  justified  by  analyzing  costs  - 
unfortunately,  only  a  portion  of  the  costs  got  reviewed  in  that  decision.  We  must  assure  that  fusine  decisions  are  made 
with  a  more  complete  knowledge  of  our  opermions.  As  leaders  of  information  services  we  most  assure  that  all  operations 
are  fully  understood,  and  that  means  a  clear  acceptance  of  a  strong  philosophy  of  quantification.  The  implications  arc 
obvious.  Once  infonnation  service  value  is  accepted  as  measurable.  ^  positioning  of  the  services,  the  infiBstrucnire  of 
the  service  organizations,  and  the  skills  needed  for  its  leaders  all  become  quite  differeni  from  what  is  cmisidmd 
iradititmal  in  information  centa  opertiion.  Given  the  significant  change  in  orientation  that  is  required,  how  does  a  leader 
begin  the  process  of  transforming  an  operation?  I  believe  the  transition  begins  with  a  philosophical  issue. 

Incompatible  PUfasophics 

Information  service  providns  operate  under  one  of  two  basically  incompatible  philost^es  regarding  ibe  value  of  their 
services.  The  philosophy  most  ccxnmon  among  traditional  infonnation  managers  is  that  U^ormeuion  organizations  are 
institutions  provi/kng  services  of  immeasurabie  value.  The  philosophy  required  in  the  modm  infonnation  center  is  that 
i/rformation  services/products  have  a  measurable  value.  The  reason  these  two  philosophies  are  basically  incompatible  is 
illustrated  in  Table  1  which  presents  the  attributes  of  organizations  opereting  under  each  oi  these  world  views. 

The  results  of  embracing  the  "measunfale  value  qtproach"  include:  the  information  center  becomes  a  pan  of  the 
mainstream  the  organization;  the  infonnaiion  center  is  viewed  as  an  infarmaiion  deJivoy  mechanism  nither  than  a 
warehouse  oi  infonnmion  waiting  lo  be  used;  the  rnsnagm  of  these  centers  work  moTt  closely  with  key  decision 
the  infonnation  centers  compete  openly  for  resources  from  the  same  pool  as  computing  ceMers  and  management 
infonnation  systems  groiqis. 

Bringing  about  the  shift  in  philosophy  requires  significant  work  si  all  leveb  in  an  organization.  One  sound  mechanism 
for  creating  such  change  is  via  implementing  a  planning  process. 

TaUe  1 

CoHBpmrbon  of  Organhntiooni  Attributes  of 
Two  Conflictiag  InfomsatioD  Vnlue  FliBooophies 

ImnMasurabie  Valiie  Approach  Measnrable  Vahre  Approach 

-  Orgaitizatioa  is  justified  by  qualitative  -  Organization  is  justified  by  quantitative 

assertions  assertions 

-  Resources  required  are  quottified  -  Resources  required  are  quantified 

•  Output  measines  are  de-emphasized  -  Output  is  quantified 

-  Productivity  is  not  (and  canoot)  be  -  Productivity  is  measured 

measured 

-  Link  between  mission  and  output  is  -  Link  between  mision  and  output  is 

aubjective  objective 

-  Budgets  giow/shrmk  increroentally  -  Budgets  are  constiucied  progremmatically 

•  Accountability  focuses  on  resources  used  -  Accountability  focuses  on  input  md 

output  measures 


CrcaUat  *  Pluudiig  Eaviroaift 

To  start,  it  is  to  document  the  ci  the  orgniaatioit  llie  missioo  statement  defines  the  present  funetkm, 

scope,  and  purpose  of  the  oegsnization  in  terms  nderstood  by  iu  memben  and  those  it  series.  Cieaiing  dte  mism 
statmnent  helps  to  develop  a  mutual  nodmtaivttng  amoof  the  managers  of  the  products/services  provided  by  the 
information  organizatiOQ  and  begins  the  process  of  change.  A  compamon  to  the  mission  statement  is  a  "visicn"  stmemou 
which  define  the  desiicd  futnre  of  the  orgsDOUiofl.  It  provides  a  verbaluatkn  of  the  hopes  and  a^mations  of  the  center 
roanagemeoL 

Next,  u  is  necessary  to  determine  the  pnatinning  the  organization  with  regard  to  populatioits  served  and  those 
organizations  providiiig  compedcive  services.  This  means  a  thorough  understanding  of  aO  products/krvices  offered  by  the 
(vganizatioa  must  be  developed.  These  pioducta/services  must  then  be  compared  to  those  of  the  organizatkms  competing 
in  the  same  arena.  Some  attributes  to  aid  in  such  comparisons  include  cost,  speed,  quality  and  complexity. 

Forcing  an  organization  to  compare  all  of  its  produce  and  services  to  other  compeliiive  products^services  in  this  manner 
provides  several  desirable  results: 

•  a  comprehensive  list  of  ail  products/aervices  is  developed 

•  (he  costs  ot  each  produc^servke  must  be  computed  and  cocTqnred  lo  other  (sometiines  commercially  available) 
competitors 

•  attributes  considered  important  to  determining  distinguishing  characteristics  (e.g.,  speed,  quality  comprehensiveness) 
must  be  quantified  to  be  used 

•  true  costs  of  producti^services  are  understood  by  the  staff  providing  these  services  and  they  do  not  think  of  them  as 
free  or  sunk-cost  items 

•  incremental  cost  and  variable  cost  cooc^  can  be  introduced  at  ibis  stage  id  imderstand  the  infiuence  of  economies  of 
scale 

In  additioo  to  understanding  the  praducts/services,  it  is  essential  to  tndeistarid  the  user  segments  served,  the  size  of  each 
segment,  die  needs  of  each  segment,  the  source  of  funding  to  sipport  services^producu  for  each  sq;inent,  and  the 
dynanics  of  diese  segments.  In  many  cases,  the  segments  served  are  not  connected  directly  to  the  funding  sources  for  the 
informatioa  center.  Thk  is  a  case  where  the  "buyers”  are  not  the  same  as  the  "customers.”  Thus,  it  is  important  to 
segment  and  usidentand  both  populations  and  any  indirea  linkages  between  the  two. 

Once  (he  mission,  vision,  productsAervices  currently  offered  and  market  or  user  segments  served  have  been  analyzed,  it  is 
time  to  explore  the  driving  force  of  the  organization.  While  it  is  not  possible  to  provide  an  in-depth  understanding  of  the 
concept  of  driving  foroe  here,  dm  book  by  Trego  and  Zimroerman^^  presents  a  concise  and  lucid  description  of  the 
concept  Fundamentally,  the  driving  force  of  an  organization  is  the  centr^  or  unifying  concept  to  guide  strategy  setting  at 
(he  highest  organizational  level  and  at  the  sidxoniponent  levd.  The  driving  force  is  derived  from  one  of  the  nine 
strategic  areas  in  three  categories  shown  in  Table  2. 


Table  2 

Potential  Driving  Forces 


Gregory  Strategic  Area 

Products/Maikets  Ptoducts/Services  Offered 
Market  Needs 

Capabilities  Technology 

Ihoduetion  Cap^ility 
Method  of  Sale 
Method  of  Distribution 
Natural  Resources 


Results  SizeA3n>wth 

Retum/Profii 

The  importance  of  the  driving  force  concept  is  the  idea  trf  focus.  An  organization  cannot  develop  the  coherent  set  of 
strategies  unless  it  has  focus,  and  the  concept  of  driving  force  provides  this  coherence.  The  strategy  at  the  of  an 
organization  must  be  compatible  with  the  strategies  of  each  subunit  The  overall  strategy  is  (he  framework  which  guides 
the  choices  that  determine  die  nature  and  direction  an  organization. 

What  does  all  this  mean?  It  means  that  an  tvganization,  once  it  knows  its  mission  and  vision  and  has  analyzed  its  currrot 
competitive  posture,  can  begin  lo  chan  a  course  for  effectively  expressing  its  goals  and  objectives  in  terms  that  can  be 
understood  by  numbers-orienied  decisian-maiKrs.  These  concepts  provide  the  framework  for  developing  a  planning 
process  ^Uar  to  any  vrfiued  business. 


Creating  the  Business  Plan 

Many  information  center  managers  would  argue  they  aheady  have  a  strategic  planning  process.  Too  often  this  is  a  long 
range  plan  generated  by  incremental  projections  of  short-range  or  annual  plans  or  budgets.  A  true  kraiegic  approach,  on 
the  other  hand,  begins  with  fundamental  questions  of  organization  purpose  and  then  builds  on  the  driving  force  concept 
In  ocher  words,  this  ^iproadi  recognizes  (he  need  far  stm^ies  devetoping  finom  the  tc^  down  and  tactics  from  the  bottom 
up.  When  this  approach  is  accepted,  it  is  then  possible  to  talk  about  developing  an  annual  business  plan  for  the 
organization  diat  involves  a  coherent  set  of  s&acegies. 
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A  »r**"***  pbn  hat  a  muAcr  of  ^-**"p^^****««  -  oiaoy  of  whi^  bave  aboady  been  dttcutinrt  in  tefint  of  ihe  ciwent  ime 
of  afbin  Md  the  cnmm  and  desited  <faiving  tece.  A  practical  ontHoe  of  the  mmial  bntiness  plaoning  process  adapted 
bom  Leza  aial  Plaoeocid^  indiides  the  components  described  below; 

Ovjmuzsiiooal  Anslyats  histaiy.  porpoae,  key  playen»  ayiiftrrot  events, 

ctfrea  stnietore  and  saffins,  historical  and  cmcnt 

trends  Bi  baidiiii  and  service  levels,  ament/toew  service 

idans,  cuncmMew  user  poops  to  be  served,  technologies 
use(^)lanned,  costs  versus  other  cooiperade  instkiiboBS 
or  composiie  of  mstinninns.  lesoiHCCS  required 
to  provide  services,  insttimional  strength^weaknesses. 

Competitive  Analysis  odier  sources  lor  similar  services,  competitive 

advantages  of  mftitptifln  ^  of  compehiors,  rating 
of  cocipetilors  IS  strong/hveragc/wealL 

Insdiutioo-wide  Analysis  cuntni  status  ctf  centers  of  a  similar  type,  projected 

future  status  in  terms  of  funding  sources,  range  of  services 
tiered,  silt  and  growth  measures,  key  growth  ^tocs,  life 
cycle  of  various  services  oflered,  financial  nends. 

Mtfket  Analysis  for  the  institution,  what  is  the  geographical  scope  of 

usen  ser^d,  demographics  of  the  users,  how  can  they  be 
segmented  (Jay  iAformaiion  needs,  locaiian,  bntding,  me.), 
distribution  elqgtnfeit  imd,  promotional  strategies  employed, 
volume  of  services  deliver^  rate  of  powth  of  services, 
share  of  maitet  served  by  center  versus  competitors. 

Strategy  Setting  specific  long-term  objectives,  performance  indicaion  to 

measure  objective  achievement,  assumptions,  strengths 
to  exfdmt,  weaknesses  to  overcame  or  avoid,  rides, 
specific  strategies  to  be  used  involvtng  technology, 
ustts,  services,  operations,  finances,  back-up  alternatives. 

Management  Aiolysis  Identify  key  functions  and  who  is  responsible  for 

each  (e.g.,  pi^lic  relations,  service  promotioo,  service 
delivoy,  competitive  analysis. 

Fuiancial  Analysis  Profit  and  loss  statement  in  terms  of  detailed  description 

of  revenue  (firom  grants,  taxes,  sale  of  services,  other) 
and  operating  expenses:  balance  sheet  listing  all  assets 
and  liabilities;  break-even  roalysis  showing  fixed  rod 
variable  costs  versus  revenues  for  various  levels  service. 

Perfcumance  Analysis  compare  lesulu  to  commiimenls  made  in  business  plan  and 
adjust  accordingly.  Report  results  in  an  annual  performance 
mpon  comparing  objectives  to  achievements. 


Coochision 

This  paper  is  on  previously  published  material^M^.i®^  concerning  information  center  ptanning  and  opcraik»  and 
builds  an  argument  for  treating  information  centers  as  businesses.  They  operate  in  a  competiiive  environment;  they  need 
measurable  ouqMtfs;  ttey  require  a  planning  process  resembling  that  of  my  business;  rod  their  opemion  needs  tt>  be 
monitored  on  a  periodic  bads. 

It  is  oyifTtial  that  a  ttiorough  evaluation  of  caneni  information  center  goals  and  operatkms  be  undeiuken  before  new 
courses  of  action  are  charted.  Without  a  knowledge  of  current  resources  (including  strengths  and  weaknesses),  otw 
directions  can*:  be  plroned  cffeclively.  The  analytical  approaches  proposed  are  based  on  sound  business  practices  and 
result  in  the  operation  of  information  centeo  that  can  base  their  existence  upon  a  measurable  retren  on  investment. 
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risk  analysis  provides  an  objectively  based  approach  to  to  describe  this  corporate  planning  process,  to  docunii 
the  problem  of  assessing  and  managing  risk.  As  a  decision  those  steps  DTXC  has  taken  to  Implement  the  corporate 
making  and  risk  assessment  tool.  It  Is  not  only  capable  of  plans,  and  to  present  lessons  learned  for  use  in  futui 
Identifying  potential  losses  that  could  be  unacceptable  planning  endeavors. 

for  a  given  system,  but  It  can  be  used  to  determine  which  RPTA:  A0>A17152S  DTIC/TR-86/19  86/08/00  87N1S902 


■  a  ? 

u  (.  V 

p  c 


V  c 

3  c  >  e 

a  0  E 

V  (.a 


C)  •i- 

0  V  c 
c  0 
♦*  a  ' 

C  E  -H 

-3  •  e  g 

«  —  3 

.  .  a  < 

tt  e  e  c 

o  b  o  o 

2  e  « ' 

z  f  o  e  u 
o  £  t.  o 
o  •  -1 
o  e  b  V 
'n.od  •  c  b 
to  bee 
e  o  « 

u  •  >  m 

L  c  ^  o  c  p 

^  n  •  m  t.  o  • 

b  3  c  c 

.H  V  « 

ae  e 
a  fi  3 
_  c  t  a 


•  C  tt  o  «  0 

J  c  <  c  b  c 

.  e  I  c  -  b  o 

-  -N.  «  3  C 

)  z  o  a  E 


>  e  I  e  c  I 
e  o  L.  0. 1 
a  e  c  c.  e 
e  Q  e  u  v  b  ^ 

V  L.  i  C  I 

e  e  3  E  e  e  I 

o  E  u  3  I 

.  -Site: 

c  e  V  e  o  I 

&N  e  e  o I 

tt  * 

—  £•♦*>  « 
e  e  «  c  o 

e  e  a  z  1 

C  •  •  1  I.  W  I 

b  e  e  p 

«  p  —  -  b  ( 
n  O  fi  Q-i  c  < 

e  E  e  I 

b  c  «  E  1 
3  a  b  fi  1 1 

—  5  c  u  o 
o  b  e  o 

•  pb  £  e  j 
£  -^  e  u  c 
o  ♦-  c  e  - 

fi  t  *-  »■ »-  c  ■ 
£  «  3  fi  *- 

V  e  E  o  e  u 

e  £  0 . 
E  E  e  o  I 
e  o  0  u 

fc.  c  b  • 
M  <»-  e  fi  c 

>  >  —  fi  • 

«  a  «  L  e 
o  c  e  p  a 
>-  0  p  >  c  I 

t  Q  —  X  ! 

fi  c  e  o  c  ■ 

C  P  fi  . 

p  u  o  0  fi  > 
-  e  *^  £  t.  t 

-  0  ♦-  PK  ' 


i  m  c**  o 
•  £  ^  c  • 
»  ♦*  b  e  *-  > 

>  E  fi  • 

.  a  u  e  u 
I  £  e  L  • 
b  c  £ 

\uu: 

E  3  C  O  O 
e  u 

>  <-  u  a  e 

)  p  c  c  *-  e 


o 

J?  . 

f  s 

e  c 

O  c  fi 

s 

”S  s 

r  o  tt 


C  Jt  E  a  £ 
e  t  c 
fi  e 

b  H  N  e 

e  e  >  e 

e  .  b  — 
e  «  m  ** 
e  c  -CM 
L  o  e  •  o 
b  o 

b  fi  b 

fi  fi  E  c  • 

—  5  fi  • 
e  ^ 

P  fi  3  «  c 
*'  m  **  9 
0  e  c  E 
V  c  o  e  e 
e  L. 

«  p^ 
t-  e  3 

a  e  c  3  b 
fi  £  0  b  e  T 
e  L  4 
o  t 

H.  O'-—: 

0-r<Cfifi- 

fi  3  1-  0  < 

e  £  p^' 

—  ♦'0-04 

a  —  L  e  4 
3  >£  e-- 
o  a  a  p  e  4 


«-  £  a 

O  U  >^tt 
-  6  < 

e  fi  c  o 
a  e  «  o 

3  a  < 

e  v>  Ni 

•  *.  3^ 


o  a  .  > 
e  c> 
a  o  - 

—  t.  .X 

fi  3  a  tt  « 

>  o  e  V 

—  a  —  ►  I. 

fifiLO- 

c  c  o  tt  fi 

<  ♦'  <  u. 

(.fin 
•  •  e  ^  CQ  . 

-i  V  o  2  • 

»-  c  p  I  o 

t  •  c 

p  o  -  <  ^ 


—  o 

c  e  e  I 

£  ec  **  I 

b  o  •  c  e  < 
c  e  >.  a  0  £ 

fi  I-  p  ( 

C  i 

-  9  m  m  m  n  ** 

a  a  e  I  e  E  c  I 

a  c  0  P  >  t.  £  I 

c  e  -  o  c  o  V  1 

—  ^ 

b  e  c.  e  c  a 

c  a  e  >  o  -  v  I 

-  a  e  ♦*  -I 

V  V  O  O  «-  I 

o  b  a  M  e 

C  b  fi  K  C  I 

>«  e  fi  c  a  a  e  I 

aa  a  p  i 
0  3a  >«  o 

-  e  £  b  ♦'  e  : 

o  e  u  u  3  £ 

b  £  u  t  +»  c  ^  I 

o  ««  3  fi  a  o 

£  5  e  -  a 

^  a  a  e  ««  a 

e  o  e  e  £  s  e  < 

E  1.  tt  p  a  • 


£  3  O  E  O 

i-«  1.  C  • 

a »-  w  e  0  a 

a  a  a  u  e 

e  fi  w  e  o  o 

p  £Pb- 

—  V  I.  ♦>  1.  c  > 
1.  0  e  3  fi  (. 

u  V  c  p  e 

a  tt  c  —  a 

«c  e  e  o 

o  o  a  £  b 

-  c  t-  e  c 

c  o  an 

t-  e  o  —  3 

o  b  -  ^  a 

P  c  4^  fi  ««  a  e 

e  e  fi  N  —  e  o 

1.  £  E  -  C  K  1. 

I  (.  C  3  3 

a  0  o  n  I  >  o 

-  o  a  E  E  a 

£  e  c  (.  o  u  e 

»-  t.  »-•  o  u  <  t. 


o  L.  e 

e  g  o 

SC 

O  E  ID 

o  b  e» 
e  u  c 

a  e  e 

3  1. 

e  fi 

b  P  a  a 
C  O  3  ID 


I  —  o  fi  ••  £ 
'a  ■  b  c  «« 
I  e  a  fi 

i  **  e  -  b 

•  '£  p  c 

)  t.  C  4^  fi 

'  4*  O  fi 

)  a  ♦*  •»-  4*  — 

a  o  n  z 

)  >  c  b  a 

1  s.  —  b  u 

.  e  £  c  o  w 

)  c  a  3  - 

>  -  fi  o  a  jr 

E  3  u  e  (. 

■  -  a*-  0 

i  —  '  fi  » 

*  e  •  o  L  4* 

•  L  0  e  4^  e 

>  tt  u  m  a  z 


3  £  bn 

—  »  4*  e 
fi  C  N 

>  a  fi¬ 
st.  E  i.  fi 
O  P  fi  ■ 
-4-  O  E  -I 
4^  U  —  E  O 
c  fi  e  3  > 

f«».  >  a  ■ 
e  m 

&e  b  fi  m 

>  L  m 
—  ••  fi  fi 
fi  4^  a 
>  fi  e  a  K 
e  4'  -  e  u 

J)  4* 

—  —  fi  C9 
C  fi  L  E  U 
fi  3  O  —  tt 
-  b-  4-  a 
P  t.  a  z 
..as 
o  >>  M 

4*  a  a  ♦*  r- 

0  2f  e  fi 

£  -  -  ao 

0  0  Pb  o 

n  b  o  3  m 

o  5  -  p  < 
(.  £  e  e 

P4<'  >  b  « 

peeCM 

a  E  b  n  H 


e  e  e  a 

aba  — 

e  o  a  0  b  • 

£  E  fi  (.  c  a 

4#  4^  fi  fi 

a  fi  C  > 

tt.4'—0—  — 

0  e  M  p  o  fi  v 

-  >  n  o  o  fi 
to  -30-0- 

0  >—  e  c  P'v 
D  fi  fi  a  £  a  e 
.03>0Ut.O 

3  4'  p  e  e  "N 

s  e  0  4^  pio 

.  b  >  o  (.  « 

p  e  —  p  e  4^ 

>  p  a  c  £  c 

>  o  c  0  «.  4^  e 

J  —  e  —  o  E  - 
e  £  V  E  e  • 
p  >  e  fi  a  o  av) 

.fikOCUfilD 

-  b  tt-  -  V  C  CM 

Em.  fi  o 

•BO-e£E- 
p  fi  o  4'  a4'  « 

>  s  c  b  g  jc  » 
3  b  e  3  p  a  < 
B—  C  b  P  -  tt 

-  *  fi  -  b  t  z 

-  <  b  fi'v.o 

-  tt  b  e  c  a  ao 

•>  -{  e  £  fi  a  c 
Dw  U  4/  3  —  n 

3  c  c  u  JC  n 

•M>.«eottfi- 

a—  o  -  -  E  « 

^  0  fi  t.  ««  b  0) 
B  —  P  tt  fi  c  < 

3S« 


»  e  fi  L  -  « 

»a>uttEe«cM 

•  —  —  O  fi  £  o 

'fivt.exttOcM 
)  >•  u  P£  e  I  fi 
» *-  e  4*  E  r*  CM 

•  •  —lb  fi  o  »  z 
.cpcot.Liur> 

i<0fi4'fittoao 


e  E  tt  —  tt  a 
tt  a  0  tt  fi  H 

3  Ce  JDbttj 

tt^- lagttcaz 
EOtt  E£Ufi  ^ 

04'eae  fi  e^tt 
w  e<— attttiB  o 
a  c  p  e  3  o  fi  • 

—  -  -  -  opcfiefiE 
-I  att  tt  —  n  L  tt  3 


.  9 

a  e  are  £  3 

-  c  p  0  0  4^ 

fi  0  03---.  0 

C  -  tta3tt3C£4' 

—  «  a  a  fi  0  4* 

E  b  >eEa— -£ 

btt  bcacttactt»tt 

ce  CfiCfi  fi-fi  fi 

e4'  a  — ZfittCttap 

v  ■  X  a  tt  c  a  tt 
tta  Bafi'3  —  see 


>  4^  Z'-CttttttZefifittfifi  E4'>.eea4'a£t 

fi  ufiiufi  u  ttttbttpbfibe  4'0  ec3i-{ 

z  -'0^4'fiabc  ufi3Etttte-eap—  i 

'»'aba£ee— a— fitt— fififitt  —  BDc  e  • 
e  e  4'bb  cfitt  ott>>.e-  bai 

tt  aaba_  — a-oa£Ctt0a  p  acD  —  -  ci 

g  ''-cattfiE*-  tt-c4'  e-b3  3>-- 

>  E  >4'<'-Ce  fifittCO  -  03*^0  -ttOjC^ 

tt  ttu  tt— tttttttt>bpafic<  fi4'eB  ttfii 

L  5  Ofittfiafifiab--  EEOO'ttttttaopE* 

I  >  tt->ecnptt>  fi3ce  -  aEEee4'  i 

-  «04»£e  3aac  -4'bab3oe  o  bci 

-*  fififi  acae£04'eoaco( 

•>fiea»-  tt33Cfi- 

e  p  ECbfoopbfitta  ttn  Efi  a—  ai 

o  fi— OfifiKO-oc—  n  -ap  e  a3-4 

c  -jzo  ^  ttSfiTPbttao  a4'ceeaoO'> 

>  e  •.«  ttfi'.fi*)  c  ojcttucae>tte—  fi] 

tt  — Utt3tt3— E  -fib  tttta.->*e-3ttttb 

3  fiwoofioott  •  z  c  9  9  o  ttapttna  c 

j  fi  c_  ^-fioco  -a3-*tt  -aoctt] 

u  oa>4'ettOtto  —  ttttCttfiaoceafittOOi 

-cvtt3^>.'C  —  tta— fiettcu->o  ci-tti 
)  ♦i'^eg  c  -ttfi3attC30UfifittD  **  * 

&  fitt-tt-OC  fiEEa  tt-04'£  ttbfi>>l 

«  z  3  tt  tt  c  -  ab  Ett  fif'fitt  e  c  cec. 
o  ^y«304'..-£s.oa  —  £a£«ettae£fittfii 

u  eo>a— fia«o«'EU4-tt  tt—  Efio  ob' 

•  «>cec  e<a  eepfi'«.a 

ovfiOfittO'C  —  tt3'4'  jea34'  oacec 
E3bvtto_a-  a  E3*tttt  aattE-ot 

ttc—  e«.b—  p>vbttttBOOtt>.>tte  e« 

\  ^  •  j  A  •  P  c  e  a- 7  a  0  o  E  tt  3  b  fi  a  n  £  IE  4 

>Z  80  —  b>fi'—  fi«>Oe  C£  3fiatt  I 

.v-  —  —  ♦*  c  04'aettc—  >gct 

•  cvj  -lb  *'eb<fi— cobo-3c>  ofittaooc 

?S  ^et'ace  c>o-e-Ctt-  fi-  3pe-i 

.E  •  fivcx*— 4'ttfi  — tt—  c—  Ettfitttttt  3a4'4 
jC»-  ufififittfifi  ettfit».oEoe—  3figaa  nc 

Sttv-boE-tt  uetteicE  se* 
•ttOttfiae— oas  — .-3tta4'  >ttC— ci.  o— ttt. 
j-a-tte-uittett»>boagae-tteoe4--30i 

•  >»>— 34'tt<^4'E<oo«>fii'  aa—  M.>ttfiav4 

•C'^«fiCECJ0C0UttttC--U3C30C«gECCl 
}  eeo-i4--wtt-u^ttft-  o  fip  a  e  B4'-£p  a  e-t 


fi  — 

£  .X  O 

•  — 

b  4'  V. 

C  r- 

fi  e  o 


tta  — 

a  p 

p  e  a 


tt  e  tt  r> 

CP  - 

-  —  e  O 

tt  >  • 

fi  u  g  ID 
MO  a  i  « 
fi  e  fi  iM 
b  b  tt  o 


e  a  —  I 

O  O  U  I 

tt>  •  44  1 


o  cx  a  tt  I 

-  e  I 

tt  ■  tt  tt  I 

fi  a  b  c  I 

c  e 

a  •  e  u  I 

C  ID  X  I 

-  ->  fi  C  I 

c  a  -  0  ' 

c  oe  <  - 

fi  <  4'  I 

—  —  P  •  « 

tt  tt  'V  L  El 

fi  B  fi  tt  < 

e  tt  tt  o 

tt  c  c  M.  < 

fi  e  ••  e  c  ' 

tt  U  •  C)  tt.  I 

o  u  < 
pc  c  -  ^ 

tt  0  Ui  o  fi  ' 

e  -  z  -  o  . 

O  tt  Hi  tt  —  1 


tt  O  Ui  O  fi  * 

fi  «.  C9  <4  h>  1 

E  C  C  I 

fi  44  .  fi  I 

tt  z  a  I 

a  -  tt. c 

>  fi  e  fi  e  : 

ID  O  H  U  ' 

—  ID  —  fi  ■< 

••  C  <  C  O  I 

-I  £  Z  £  I 

K  U  tt)  0  fi  I 

H  e>>K  fi  £  ( 

3  K  «  tt-  h>  1 


UTTL:  Information  resources  management  in  the  R  and  0  AUTH:  A/VANAUTRY' 


«  0 

0  6-*“ 

E  >  •  -H 

♦*  —  V  e 

3  «-  W  O  E 

«  >■£  i. 

•t>  C  «  o 

o  •*- 

H-  •  £  C 

-  o  • 

£  •*-  3  « 
V  >  £ 

—  O  "O  V 

i.  t/>  9 

•  e 

n  n  ■  c  ■** 

E  3  n  e 

e  EE- 

e  e  e  • 

M  £  4^  ^  C 

>  n  a  o 

n  >•  E  4- 

4-  e  * 

c  c  (. 

0  3  e  c  « 

O  £  «  £ 

4^  U  -H  «  t- 

«  O  £ 

^  E  «  «- 

4  (.  O  >*0 

)  0  O  «“  C 

H.4'WC- 
J  C  C  W  -  E 

-  ^  a  « 

U  E  C 

)  a  a  u 

)  (.  3  e 

-  3  a  fl  >  (■ 

)  4«  4^  «  « 

4  3  «  £  « 

•*■  o  £  3 

)  4-  4^  W 

JO  **‘0 

r  c  e  c  c  e 

^  o  0  «  o 

J  C  C  EC 

j  c  a  «  o  e 

)  «  4-  a  >■»• 

c  t>  O  4. 
a  o  4.  a 

L  a  u  a  a 

c  E  a  E  X 

r  4-  -  T3  -  « 


•  e  D 

>  H-  «  e 

e  o  u  > 

£  u  0 
>•  e  4.  c 

—  V  a  0 

^  ^  .e- 

—  —0-^4' 

3  **  4B  e 

£  —  C  U 
>■»-<£- 
e  «  c 

£  M  •  ts  3 
4«  a  4.  -  E 
u  a  3  E 

•  U  M  O  0 

c  e  3  u  u 
o  a 

■^at3«*- 
4*  £  a  E  a 
«  4«  u  a  4^ 

4.  c  4< 

4<  o  a  «  c  ^ 

a  c  •«>  >  a  o 

c  «-  4.  a  a  00 

a  4^  a  3  00 

Q.^  Q.  C  V  C4 

X  o  a  z 

4.  *-  a  •»-  £  <0 

a  u  <  4^  00 

4-  a  c  «  c 
^  o  E  0 

a  c  4. 

£  a  o  4^  g 

a  a  <4-  a  o 

a  o  £  c  4.  \ 

a  4-  —  a 

4*  «♦-  ao 

a  4.  >.t3  O'-s. 

o  a  £  a  0  40 

4->  N  U  00 

occ- 

4<  —  o  —  L 

--  a  a 

4-  4->  4-  a  4^  . 

c  c  a  t}  4^  a 

a  a  E  a  a 

3  E  4.  £  - 

a  o  a  4/ 

a  ^  <4-  a£  — 

E  a  c  o  o  4. 

a  E  -^  £  3  0 

4-  -  4-'  o  £ 

a  a  4.  V 

>  0  £  0  £  3 

a  v  41*  4-  4-  a 


o  3  4/>< 
tJ  I 

a  o  I-  c 

3  4-  •-*  W 

-  au. 
a  u. 

>0  4- 
C  04  • 

a  a  C9  o 
£ 

4'  a  4D 
a 

oi  u  a 
c  -  ••«- 

4.  L  »  -> 

3  a  > 
a  a  •  X 
a  Q  o 

a  •  o 


I  o  a  c  o  a  : 

a  a  a  4 
.  o  c  a  4.  « 

I  c  o  a  3  £  - 

.a  -  I-  a  o  t 

4^  a  a  3 

•  04  a  a  a  c 

E  a  E  I 

•  a  4.  -  ^  - 

>  o  Q  a  a  I 

c  «-  a  >  c 

i  a  c  4i>  a  o  « 

>  —  o  £  —  t 

•a  3  a  I 

I  Q  >-^  ••-  O  4-  a 

»  ci  o  o  a  a 

O  00  4.  £  «.  4 

)  c  ■  a  a4«  o  4 
:  a  Ok  3  4.  fl 

-• «-  t)  a  a 

a  a  c  4/ 

)  U  A  >  a  4)  >.  I 

'  a  Ok  a  £  4 
[  ar-  a  a  < 

I  a  00  £  a  o  £  ( 

c  o  r«  4>  u  4.  a  < 

4-  •  —  a  a  £ 

:  a  40  a  >  3  « 

•  <«££.£  f 

z  -  a  u  c 

«-  c  a  4.  o 

o  u  3  a  —  B 

lu  a  -  9  **  t 

)  4^  40  a  a  a  a  0 

:  4.  w  e  E  a  E  < 

a  E  a  4^  1.  £ 

-  £L  a  4*  o  - 

>  4.  a  4.  «•.  ( 

• to  o  >  3  c  £ 

•  a  4-  «i.  to  o  —  e 

-  4.  •*. 

•  01  £  C  £ 

4  a  a  o  o  •*■  a  • 

>  4-  a  a  o  £  4 

)  c  E  o  >*  L.  I 

E  4.  a  a  o  - 

a  &  E  44  u  4 

c  4.  a  c  —  a  c 

a  o  a  o  3  L.  c 

o  ••-a  - 

a  4.  c  c  a  ••>  4 

•  a  o.  <  —  fit  0  « 


>  a  £  >«t. 

-  £  u  H> 

:  4^  3  a  o 

a  -  c 

-  4*  — 

)  £  £  a  4* 

:  a  a  a  — 

■>  4*  a  3 

a  3  o  a 

j  •“  4>  a 

)  —  a  1. 

)  E  £  - 

.  ^  a  a 

a  •4'  4.  3 

)  a  £  a  — 

:  a  o>£  a 

j  4^  > 

£  E  O 
■>  c  — 

)  a  4^  4.  a 

»  -  o  ^ 

£  4> 

'  a  £  «-  c 

*  #  o  -  a 

4.  --  a  3 

'  £  a  cr 

.  a  3  a  a 

>  —  c  a 

£.  o  c 

-CO  o 

;>  o  «4  a  u 

)  4'  a  a  a 

:  a  a  u  £ 

}  E  a  3  >- 

c  o  £ 

D  0  Q.  a  - 

:  C  O 

•  C  3  O  4^ 

'  -  a  4*  a 


4-  a  - 
a  4.  c 
£  a  a 

4*  o 
a  4. 
a  a  o 

0  3  91 

c  —  a  Ok 
^  a  »  r- 

>  >  4.  00 
a  a  cv 
a  4.  a  z 
a  3  (0 
a  £  00 

£  4.  a 
4-  o  £ 


£  O  O  - 

a  >  a  o 

L  I  k4  V. 
3  «4  <0 

a  -  a  00 
a  a 
a  a  a 
E  3  3 

>«  c  a  > 

<-  o  >  4- 
^  -  c 
O  4^  —  — 

-  a  a  o 

4'  E  -  O 
L.  L.  4’  a 
a  0  r 
0.14  a  • 
c  3  c 
a  —  £  t 

-  as 

E  a  4- 

a  0  c 
a  4.  o  c 
3  >4  0  - 


£  £ 

.  £  0  4- 

)  c  - 

-a  £  c 

3  - 

«  ft 

-  a 

•  a  a  4^ 

}  u  a  c 

c  o  - 

9  a  L  0 

>  <4  ---  3  a 

-  a  04  0 

1  o  00  a  c 

>  I  9  o 
£  91  L  — 


r  0  o 

a  9k  —  a 

>  o  r-  4*  £ 

:  a  00  a 

:  a«M  e  4^ 

I  a  I  4.  a 

t  O  49  0 

4.  00  14  4< 

:  a  z  c  L 

•  <  -  0 

UJ  Q. 

•4  iu  a  0. 

O  t/>  £  3 


a  3  £  4>  a  <4  4. 

4^  c  a  00 

a  •  a  &  Ok  «4 
O  <  «“  > 

U  Ok-  u  c 

-  c  a  a  a  0 

a  4-  UJ  L  a  3  Q 

£  k-  w  O  E  0  3 

i~  UJ  a  E  a 

4X  c  4*  a  £  £ 

■ -fit  o  c  4.  a  a 

-J  <  £  «-.  o>  — 

K  CD  c  oa  — 

K  >v  o  c  u  £  a 

s  <  -J  a  a  K  u 


a  a 

q.*4  £  a 

-  a  >*'  c 
u  £  a 

c  c  «4  u 

a  a  -  0 
u  a  >  £ 

£  —  a  o 
-  a  0)  E  - 
a  c  L  £ 
a  t.  -  a  3 
a  o  a  4^ 
u  a  c 
o  a  a  c  o 
L  0  4-  -  - 
a  c  u  4J 
L  a  c  4^  e 
O)  a  -  c  E 
c  a  a  (■ 

-  4.  -  E  o 
**  £.£  a  *4 
a  c  c  c 
jc  a  a  —  - 

4.  £  <4 

a  4^  a  a  a 
E  a  L  £ 

.  -  o  4- 

c  a  u  4. 
o  Q.  a  a  (4 

-  O  *4  0 

4'  U  4’  «4 

u  a  a  0  > 

8  4.  £  >  — 

L  o  —  a  — 

4.  a  c  £  a 
a  £  -  4-  3 
I  4'  £  o 
c  4  £ 

Ok  >-  -  u  £ 

-  £  3  -  C 

a  £  a 

a  £  c  3 

£  a  o  a 
L  '  I-  -  0  a 
£  3  4-  4  a 
u  a  a  c 

4.aE4- 

aa4.c- 
a  E  0  a  a 
a  «4  4  E 
aacx- 
u  £  -  a  4' 


c  a 
a  a  a 

Q  a  I 

4.  c  a  a 
a  a  4. 

c  <4  c  0) 
UJ  a  0  o 
£  -  4- 
4.  a  4>  a 
a  £  a 
3  -  e  Q 

•4  0  4. 

a  o  £ 
E  o  *4  c 
3  a  c  a 
4.  -J  — • 

4/  I  fit 
C  C  <4 

a  -  o  a 
N  a  u 
a  4J  a  c 
c  a  3  a 

O  C  -  <4 

—  a  a  a 
4*  o»>  o 
a  01 

E  UJ  a  £ 
4.  £  c 


o 

c  a  c  4.  4 

0  3  a  a  < 

-  00a: 

^  jt  -  -  3  t 

a  4.  4.  «4 

E  o  a  -  4 

4.  3  >  c  c  i 

o  a  Ok  a  ( 

•4  e  *u  -  —  < 

c  a  o  a  3 

-  4.  o 

•*.  4-  ^  —  4 

£  c  a  4^  < 

a>3—4-- 
N  £  O  4J  a  4 

-  a  u  a  Q.  < 

-  4.  4j  a  I 

a  a  a  a  c  4 

-  £  —  4 

<J  3  C  4.  «• 

a  0  —  £  4 

Q.  •  aa- 

a  c  a  £  4. 

o  ak4J  a  T 

>-  C  £  4 

-  4J  a  o  -  « 

£  a  £  4J  a 

Ok  N  (J  c 

-  -  Ok  o  « 

'  £  E  >  C  U  < 

J  -  —  —  4 

J  I*.  4.  **  3  a  4 

o  a  c  o  £  : 

k  o  a  ( 

■4*  4*  •  o  4 

c  «4  a  a  4^  < 

)  a  o  c  4.  4 

k  E  00a 

-  a  J4  u  4^  a  < 

>  Ok  a  U£- 

t  a  a  >  a  i 

c  4'  a  *>  Q  < 

3  a  E  c  - 

>  E  *'  Ok-  - 

:  c  a  c  3  < 

4^  a  c  -  o  : 

J  c  c  o  u  —  < 

jaa-c— 
k  -  E  4«  a  o 

O  4.  -  3  ^  t 

-  a  £  -  t 

L  <4  a  c  «4  a  < 

«>  o  c  £  < 

r  UJ  a  u  —  4  4 


i  a  E  £  « 

k  u.  3  c  3 

L  -  c 

•>  £  c  4 

'  4-  a  o  4 

•  £  N  4  4 

!  c  <4  a 

k  a  0  c  4/  « 

o  a' 

L  >  a  -  E  i 

:  —  _  ®  ♦ 

k  a  a  a  4 

k  4'  E  E  4^  c 

a  a  L  a  »• 

k  E  X  O 

•  -  a  4  a 

•  X  C  £  t 

0  a  -  4'  4 

>  4.  £  £  4 

'  C.  4J  0—1 

a  a  N  4 

k  a  >u.  4.  * 


-  £  £  4 

•  c  a  4^  j4  f 

:  0  -  < 

)  a  £  4»  t 

•  4  L  c  a 

'  3  4«  «  e  • 

}  £  a  a  '* 

•  3  >£ 

■  -—044 

;  a  -  a  a  4 

•  .X  -•  E  Z  - 

"  —  4.  4 

X-  •  a  Jt  : 
)  ■  >c  —  - 

—  —  a  4 

.  0  -  •-  >  B 

3  £  a  0  £ 

0  0  &  - 

4  e  -  o  I 

1  4  —  a  4 

J  £  a  —  —  - 

'  a  ££  014 
>  Ok  3  4.  4 

9  -  a  Q  a  « 

:  0  c.  a  c  4 

•  a  acK  ui  ( 


-EE  £  « 
J  a  4.  4 

:  a  o  t 

-  a  <4  •».  >s  ( 

_  -  C  o  £  4 

-  £  —  - 

3  ^  C  £  : 

-  £  -  o  a  1 

-  c  a  —  4  t 

a  a  (.  4  a  4 

-  a  4. 

D  >  c  c  4  t 
1—  a  —  a  ! 
3  a  o  E  3  - 

3  0.  a  — ' 

J  3  V  a  —  J 

_  a  o  a  •-•  ( 

0  c  a  £  • 

3  o  a  a  I 
3  —  a  —  c  4 

-  4  o  a  a  - 

•'a  t  4  < 

D  E  a  4  c  4 
:  c  £  c  e  : 

o  4  *  o  T 
3  •:  o  4 

:  c  c  a  £  I 

8  -  -  £  4>  < 

4. 

-  3  a  £  a  4 
8  O  JC  C  a  4 
:  £  4.  a  a  4 
3^0  a  i 

-  £  3  ■  4.  I 

■'  C  M  4 

8  a  C  14  c  4 

;  o  u.  —  T 

5  >  c  0 

;;  c  a  £  o  4 

-  4.  4.  a  4  t 

a  o  a  a  4 

-£»-OE- 

-  W  L.  —  I.  I 

:  JC  —  >  o  I 

4.  4.  «.  I 

-  0  •»-  a  c  3 

K  3  o  a  -  : 


a  a  a 

E  a4  a  ( 

E  >.  a  c  i 

a  £  4  £  0  * 

4,  4 

:  Ok  0  a  a  a  I 

k  o  £  £  a  ( 

u  4  >  4 

)  a  >•  £  4  I 

I  -  4  3  1 

O  -  O  £  £  - 

a  a 

k  £  —  a  a  •  ( 

•  c  4  r  a  a  * 

'9  c  3  a  E 

k  a  a  Q.  3  T 

•a  c  o  E  -  I 

o  a  0  o  4 

)  a  a£  o  > 


E  0.  3 

-  fit  - 

o  a 
4.  u  > 

o 


a  a  >  I 
4a*  I 
OB  C 
3  —  UJ  • 

a>-'^- 
z  4.  z  a  4 
a  k-  o  I 
•-  a  3  c  4 
-I  <  a  4 
»-  £  O  L  < 
k-  C  ''K  u.  4 

3  a  <  — » 


O  -aoC» 

3  £  C  O  t 

£  o  4.  a 

c  u  0  4.  a  4 

a  at  o  4.  u  : 

a  3  c  ( 

a  — '  a  L  o  a  • 

O  P4  4.  0  3 

a  00  a  4  o  —  < 

a  •  0  4  1 

a  9)  a  4  c 

o  —  a  I.  o  -  ( 

4.  a  a  ^ 

a  A  a  £  4.  4  I 

<  A  £  E  a  c  1 

r-  3  £  a 

4  tt  a  c  E  £  1 

O  A  4  3  -  ( 

■  a  a  c  >  1 

4  (9  T9  £  a  < 

4. »  4  a  < 

a  z  c  £  c 

a  o  4  t-  a  I 


a  4/3  a  a  a  a  « 

u  w  E  a  4  £ 

01  4.  3  a  ( 

a  a  o  a  £  a  ( 


>  4  a  L  (0  T 

—  a  0  •  L 

a  4  0  i 

4  jt  a  n  4  I 

a  £  o  a  c 

£  0  -  —  -  i 

4  4  o  a  Ok  3  ( 

L  a  o  a  4 

4  o  4  4.  —  c  t 

C  k/3  c  o 

a  o  a  c  a  ( 

4  •  o  >  £  « 

4.  4  —o  - 

0  —  4.  o  a  a  : 

a  a  >  4  c  I 

E  a  £  a  -  3 

—  £  c  4  a 

a  3  a  c  E  3 

a  a  a  a  0  4 

-  C  4  3  4  £  £ 

C  &  4. 

4  £  a  —  o  a  I 

—  o  E  c  a  a 
-a£Ea- 

£  £  Ok  O  -  £  l 

c  3  a  a  4  1 

a  c  4  a 

a  a  a  4  T 

a  c  E  4.  L  0 

4.  o  a  £ 

3  -  a  u.  4  £  « 

444a  o  c 

aaacca- 
a  o  £  a  o  a  4 

0-4.  c 

9k  —  4  4  £  c  ( 

0  a  o  a  - 

4  a  £  a  I 

a  E  c  3  a  i 

a  a  a  o  £  4 

X  >  —  —  O  Ok 

-  -  £  04  3 

a  a  o  c  o  c 

c  4.  -  a  4.  : 

a  a  a  a  —  £  ( 

a  a  a  4- 

X  £  a  a  4-  JC  c 

a  o  -  o  a  a  i: 

a  o  o  a  a  ( 

a  a  £  4.  a  L  I 

-  4.  3  a  a£  4 


9 


B-5 


?N  U.  « 
O  • 
-  C  C  D 
jc  m  m 
•  09 

•  a  c.  9 
•  c  o  —  ♦* 

c  ^  1.  ^ 

»  v  TJ  » 
g  ^  0  •  • 

L  C  V  T]  > 
O  •  C  3  b 

«  V  • 

c  c  «  > 
^  ^  «  0 
a^'  «a 

?c  c  c  c 
•  •  o  a 
«  e  **■ 

V  £  V  a 
c  c  «  «  c 
o  •  N 

-  —  t 

**  m  D  C  0 

«  C  4  •- 

u  4  a*- 

^  4  b  O 
C  4  O  b. 
3  C 

E  4  £  ««  • 
I  O  C  < 
O  O  b  4  </) 
U  4  4  >  Ui 

4  4 

4  4b  4  «>  IQ 
£  0  4  4  C 
•b  b  a  4 


—  O  C  UJ 
4  £  4  4  Z 
4  a  >  O 
b  £  4  «• 

4  o  a  • 

0  4b  a 
4  4  4  4^ 
4  3  4  Z 


i 

I  3  >> 


iP** 


3  £  4  O  4  V  4 


b  4  •  4 

4  «-  4 

«4  m  **  4 

—  I  ♦*  4  4  ♦< 

4  «  £  4 
•  C  ■»■  • 

4  4  ««  O  ** 

b  V  C  V  ^  C 

-  -  o  • 


4  4 

b 

?> 

•S 


b'  4 

4  a 

c 


>  4  T)  4 

a  a  4 

C  t)  4  4  E  ^ 

4  C  £  4  0 

4  V  jc  4 

V  4  4 

C  «-  5  >  3  C 

4  4  C  4  b 

b  4  £  4  4 

4  0  ♦«  £  ♦< 

*•  B  b* 

4  £  4  4  4 

c  5  •  a 

•b  o  *-  £  c 

C  £  4  O  C 

4  b<  • «-  0 

£  E 

u  0  4  4  -b 

4  4  £  i  4 

4  4  b  O 

>  O  0  — 

£  b  <b  «-  C 

3  C  0  C  3 

‘LOO  ^  e 

4  4-^4  i 

b  -b  4  a  0 

JCO 

a  V  b  4  JK  4 

4  O  b  4  £ 

b  •«>  4  4  -b 

>  C  4  4 

C  4  T) 

O  C 

—  4  4  C  4 

4*  -  £  b  4 

4  £  b'  4  a  > 

E  4  4  4  b 

b  -  £  C  4 

O  •  O  *>'  4  b 

«•  4  ■*>  a  i  £ 

C>£C-- 

4  4  4  -b  ^ 


C 

b 

O  £  C 


s*a 

O  4 

zs? 

4  ^  • 

4  o  a 
ac  -  r* 
a  o  • 

3  -  o  « 

4  b<  4  Z 
4  b  <0 
E  £.« 

4  b 
b  O  V 
4  «-  O 

UcS 

a  ov. 

4  E  Z  o 

E  4 

0  4  « 

(/t  C  ^  4 

O  4 

C  **  ** 

4  4  4' 
>044 
^  TJ 
a  C  4  4 

2  *  c 
c  I  ^ 

4  1  4 

£  O  O  b 
O  V  4 

4  4  4  4 

•^  £  C  b 
V  O  4 

4*  a^  S 
b  C  4 

4  N  a 

4  >  •-  C 
C  O  C  4 
'*■  b  4 

a  g  ar 
c  E  b  c 

4  «  O  4 


4>*^  4  4  £ 

4  a  O  M  c  b  4 

c  a  4 «  o  a** 
o  <  a  <  4  o  «> 

^044  ♦•C^'b—  I 

♦-<0*^44*-‘^  J 

4  b  £  o  C  i 

UC4nV‘«>  •>«44 
Co££Vl 
OC  (^V3^  ^  i 

•^3  »b«0|4«  < 

•bi  or* 

S  O'^-b  i  4  t  _ 
bOEb«4  «>4bi 
0_34ZCp4-V 
b  O  -2^0,  4  E  0.  4  4  I 

C4a  tMCbOBibi 

4  4  4  (/>  «>  b  X 

b  b  flo  b^  b  ac  4  u  jt  I 

o  a^-'  a  4  4**  **  b  j 

«-4  4aSb  4V04 

b4b  bCQ4<-> 
4C^CttOHi4  04 
oph  S>.|i 

4  4  CO*>gQ.4« 

404C4<»>»-EC4b« 
CK344  444 

4b  Ifzxaa  I 
4Z04i£  4434- 

**  44b4 

b  ••  •  b  £  4  b  ^  £ 

4(b  >>Cabb^4bb 

att  g  o  o  o  £  4  ^ 

4oc<^b«.c^b>ri 


£  4'-' 

4  4  O  Ci 

«■  b  4  0  4 

b  -  a  •  *• 

4  0 

b  C  O  4 

D  M  b  O  C 

4  0  4  4 
4  a  a 

CL  4  i 

OC  •  <b  0  O  0 

o  ^  O  r«  b  b 
U  ‘D  <  4  «• 

C  b  0  4 
C  4  b  0  V 

O  -  4  Z  £  C 

4  4 


4  E  •  £ 

£  4  b 
b  O  W  4 
4  4  4a 
—  O  •  •  —  -^Sr- 
4  >  4  4  0  90 
O  b  4  **  b  •*  Ct 
•*  b  a  C  Z 
C  b  g  4  £  0 

£  0  E  b  b  O  0 
O  b  4  4  <b  4 
4  X  E  4  b 
»*  4  4  b 

-£.5 aS8 

4  4  4  b  b-v 


?a  44Ca  344^ 
ui  •*-  0  V  b  i  o 

iUf-UJbO<^C4040'^ 


£  Z  V  I 
b  0  “ 

b  tt  *-  O 

|»cfl2»5*  .* 

i  4C4  b'-'Oba 

4«E<-*  ai  b4c 
b  I  4  tt  £  a  >  b  •* 

a»-  a<b  4  o  4  b  jc 

0^404 XCC£4b 
bZC  Oo44b£0 
£.0  44CbEa  0> 
'2  3  44  <  »• 

‘  E  0  b  r 


E  3  4  4 

'  O  E  4  4  C  4  E  ' 
•■(n4>(S3b4l 


it 


—  ^  -  ab  E  b  c 

►-»!»S|J04bb- 


4  «•  **  4  C  O 

b  O  4  b  ■**  C 

>  4  E  4 

4  4  >>  4 

^>-94X4 
»“  O  4  ■* 

4  b  4  C  • 

O  £  O  4  > 

«*  ab  b  O 

b  C  b  C  0 
**  **  4  4  O  4  a 

b  P  TS  E  b  ar^ 

O  C  3  b  <0 

4  0  T>  CM 

V  £  g  «>  4  4  Z 

C  X  C  C  4  £  0 

4  4  *  ^  C  b  0 


E  •  w  4  0  *> 

b  *3  b  w  b 

O  4  a  0  4 

b  >  a  4  a  4 

C  *-•  TJ  b  c  b 

-  ac  ^  c  a  **  3 

0  b  M  £  0 

b  z  £  b  b 

4  >  C  4  •* 

0  4  4  4  4  C 

4  «  »  I  0 

JC  Z  4  E 

8-v  >  4  4  £  b 

0  •  b  O  4 

^  o  c  a  b  E 

U  O  0  4  b 

b  *-  b  b  o 

4  •  .  b  4. 

JC  O  T)  4  4  C 

4  b  E  a  £  <- 

E  --lb 

*3  0  t)  a  4 

>  O  b  C  C  £ 

O  •  ^  C  4  b 

••  Z  —  Jt 

-  a  £  K  4  £ 
g  0  3  b  EC 

£-1  0.  O  4  4 

4  »  4  4  E  O  T> 

£  I  O  3  4  *•  4 
b  K  4  —  b  —  £ 
K£4  4  0** 
-•  a  4  >  a  £  b 

^  U  O  O 

K  0  b  4  P  4  4 

•-  "v.  4  £  C  £  4 

a  <  <  I-  4  I-  P 


4  0  b  3  4  b  0 

••  *•  4  b  C  3  0 

>  b  P  b  4 

4  4  4 

a  E  4  b 

w  b  C  4  b  0  • 

0  0  4  0  P 
b  b  O  4  4 

4  C  4  b  4  4 

JC  «-  4  >v  C  3  4 

4  b  O  4  3 

E  4  4  •*  i  4  O 

£  g£  4 

>»-  4  4  O  b  *• 

O  b  —  P 

•>  •  0  E  4  P 

-  P  4  3  >  C  O 

0  4  4  4  4 

£  C  O  £  P  - 

b  b  >4 

-  E  £-^  P  b 

4  4  C  4 

>  X  £  ■'s.  4  —  b 

4  4  O  E  4  4 

"  4  4  E  0  S'  4 

4  b  4  b 

b  4  4  E  b  C  4 

4  4  3  4  O  JC 

b  -  -  4 

P  b  4  b  b  E 

4  0-  —  4 

E  b  4  b  C  E  > 

b  g  O  O  -  b  O 

4  4  -  O  — 

b  u  £  4  4  b  — 

C  -  >^4  £ - 

-  a  b  3 


—  o  • 

4  a  b 

b  c 

!4 

"it 

C  4  > 

4  b  4 

4  b  P 

a.  c 

C  4 

£“r 

b  p  O  • 
4  C  b 
E  4  4  4 
b  4  0 
O  tt  4  C 
b  4  4 
C  4  £  b 

"  E  4w 

b  4  O 
O  b  4  4 
4  £  C 
4  a  4  - 

3  0  4 

—  P  b  0 

4  C  4  • 
>  4  <  b 

3 

4  4  b  0 

£  O  O  • 
K  4  b  > 
£  - 
••  4  £- 
-J  O  3  - 
b  O  3 
h-  4  b  4 
0*02 


4  4 

b  >  4 
4  4  £ 


b  b 

X  4  P 
4  E  4 
b  b  C 
C  O  3 

0  b  b 

0  c 


’  E 

)  b  s 


b  £ 
C  b 

-  4 

4  £ 
4  C  U 

Hi 

-  4 

-  b  0 

h: 

b  b  C 

■oJ 

JV  J 
>  £ 

HI 


3  >  4  E 

4  £  O 

JC  C  C 
*•>40 


b  O 

4  b 
£b 


J  -  —  —  £  4 


C  C 
0  4- 
4  >  O 

C  P  C  4 

-  4  O  P 

4  a  O 

bp  4 
C  3  4  £ 
O  *^£  b 
O  b 
4  b 
b  £  >  0 

-  £ 

O  3 
£  b  O  4 
O  4  - 

—  b  —  > 
£04 

**=^2 
C  4  3  b 

b  C  >  — 
4  O  b 
E  -  -  C 
b  b  O  O 
0  4  —  - 
b  E  b  +« 
C  b  C  4 

—  O  f  b 

b  £  C 

4  C  b  4 

£-34 
b  4  4 
b  b 

>  •  b 
>b  O 
b  -  4 


4  - 
£  b 
b  4 


—  4 
O  4 

£  4 


O  b 
b  4 

£ 
C  b 
-  0 

4  4 


0  b  4 

*  C  P 

b  4 

3  b  — 

0X4 
4  O 

b  — 
O  4  b 
£  4 
I  4  b  E 

5  -i 

b 

4  4  P 
£  -  4 
b  3  C 

O  — 

•  —  b 

JC  b  4 
b  b  b 

O  4 

3  £  > 

4  C  b 

8'g 


.  °i 

4  4  C  b 

!2fS^  .  _ 

-  b  b  o  —  3  4  - 
•  4  -  6  —  b 

’  4  E  4  -  Z  £  4 
4  4  b  P  3  4  — 

b  —  O  4  •  —  P 

4  ab  4  £  4  — 
3ECC0b44 
P  4  —  —  4  >  4 

b  X  a£  P  4 
4  4  C  C  3  4 

£  4  —  —4b 

P  3  4  4  £  — 


'5 

b  b 

O  4 

P  P 

4 

E  4* 


C  3 
>  b  O  4 
•  O  -  C 

'fh 

4  O 

3  C  S 

r 

4  i 


4  4  b  O  b  b 


4  4 

O  E  4  b 
>  b  b  — 


a 

3  b 


lu 


ab  o  4  E 

C  —  P  4  4  >  i 
ab  4  4  >b  4  £ 
-  C  4  £  4  C 
4  4  4b  P  4 

4  5  £  b  C  >  b 

“  0*  4  0  4  4  3 

>b  b  E  b 
b  -  £  -  3 

•  O  —  a  4  4  4  b 
b  4  —  b  U  P 
4b--4-4b 
>4b££b4U 
O  £  b  Qb  4  C  4 

1  4  -  0  i  b 

b  3  3  £  3  4  — 

O  C  I  g  4P 

•  4  -  b  4  >£  b 

'  4  b  b  O 

b  >  C  —  b  — 
4£4g4  —  b4 

p  0  JC  4  o  g 

l^S"8ScS 

b  b  P  OP 
44b4C£  —  — 
£  b  4  0  ab  > 
bbC4  —  —  44 
4  O  £  4  £  — 

0  3—  —  b  %. 

-b40^C0 


4 

'  b  £ 
}  Ob 


•  34 

>P  . 
O  b  4  -o 
7^44 
E  —  —  b 
4  4  >  4 

5  ®  3 

4  O  £  4 
b  •— 


4  4  C 

£  b  4  4  < 

O  >  b  I 

*  4  b  I 


b  •  b 

o  a  4 

Cb  Ui 
>  •  UI 
L  C  —  0 
4  O  4  >- 
b  b  b 

4  a  a  £ 

b  C  4 
O  —  b  0 
9  £  C 
4  0  4 
C  4  4 

O  b  »  C  4 

“S  ■•I 

P  C  ^  4  4 


-  i 


C  O  I 


4  b 


4  4  —  b 

--43 

b  £ 


S  i52£ 

-  4  P  O 

b  C  4  b  b 
4  4  —  —  — 

eS 

_  -  _  4  C 
P  -  - 

4  4 

4  E 
4  4b  •  C  b 
4  0  b  O  - 

"p'-Sb’" 

>-  4  4  4  0 

a  b  b  C  E  b 
O  O  -  4  b 

-  3  4  E  0  4 


—  —  —  4 


>=2.  £ 
•l5E._ 

“  4  b  O 

£40 

b  > 

4  4b 


4  O  O  £ 
£  4  b  b 
X  b  £ 

4  b  P 

4  0  4 

4  £  4 

3  C  3  C 

O  4 

b  O  4  E 

b*'E* 

4  4  b  4 


4  — 

O  O  C  • 

4  £  4  b 

£  b  t 

4  4  4  4 

£  -  £  £ 
C  H  — 

O  4  b  4 


—  -  O  4  P  4 

—  4  —  ‘0  —  b  .. 

P  C  >  >  4  B 

444bO0£b 
4  b  b  4  0  '  3  4 

—  b  4  OC  £  4  b 


E  4  4  4 

—  b  b  C  0 

>  b  4  —  a 

4  4  4  —  f* 

c  b  r  “ 


§|5S 


-  z 

it 


C  4 
4  4  E 

b  E 

4  4  4 

4  4  b 

c  a  0 

-  C  O  b 

?  2i  . 

4  b  0 

a  4  a  4 

?4  E  -» 
4  b  P  0 
4  Z  O  C  b 
b  4  £ 

4  C  C 
b  4  b  tt  £ 

4  a  0 

—  b  4  b 
b  •  O  g  4 
C  C  4 

4  -44  4 

-  b  3  b  4 
O  -4b 
4  •4a 

a  >  b 
b  W  p  c 

O  4  C  t 
b  £  4  O 
4  b  »-  4 

P  -  4  b 

4  4  3  0-* 

4  K  tt  4  CM  4 
C  <  £0 

••041  b 

c  a  <  0  a  0 

0  OC  b  — 

-  o  c  4  c 

b  u  <  D  0 

4  a  - 

E  b  c-  b 

b  0  0  £ 

0  .  .  c*  - 

b  —  — »  I  b 
C  4  b  0  O 
-  •  P  4  0  4 

a  C  O.  Z  4 
4  4  4  P 

£  O  —  UJ 

b  4  »-  b  4  UJ  b 
£1-  4  b  0  4 
•••►*£  aw- 
^  O  <  b  4  b 

b  b  tt  4  b  ££ 

b  ^>^2  c 

a  4  0  wM  0  < 

z 

b  0 

3  0 


RPT0:  AQARD-CP-3a5  lSBN-92-a3S-03B9-9  A0*A166641  86/01/00  DPT#;  A0-A165640  85/00/00  86N28779 

86N28793 


B-6 


^  VC  0  2  • 

§t.  —  i 

iSh 

I  L  C  «  V  ^  10 


•  O  «  f  3 

c  %-  >.2  u 

O  •  C  •  0 


(.(O  O  60'  8-^0 

0  LC0V  «-^IO 

£  0  •  Qm  mci 

0  u  u  •  ^  U>v 

•  g0£«A^ 

00*'O  3««U  2  6  £  O 

0«0U  Q’o'QmOI  O.'v 

£0C3  gii£i>c0« 

0  •  C.  >  C  i  A  ♦'w  0  L  «D 

4i*oe0io  et  0  »-p 

‘8  0  >S 

>>  O  w  2  3V  i)  ‘D 

t.  00<<0£^i 

0V<H>  8k  i0£0 

c0(.a  cO'-'Ccv  • 

£UQ0  OV  00003 

<*•  g*-  «•  3  C  %-  >.£  u 

•-—00  <0  *00^0 

Aoec  0  -Cm 

C.3  £  Ofvo  0VC 

oo-ao  u£0— >  o 

V  (OiM>CV0CQ0  0 

V(0»-o5  C.3O00- 

0C3  f‘000  —  00 

S—  Cl  0  —  VVQ—  30 

O0OOk0  C0CC  0 

V3*--  -  c.  >M-£  go  m 

0  -V  a  >UE>0 

0«vOUO0O-£ 
0a.ciac  —  —  40H>uv 
aMO(O0O«-  _  U4 
CO  v«^C-fg£0v 
O  U  3C«<.£v£CU*>0 
—  <|QQCV0  —  «.0 
V  •  «  g  0  0  £  0 

8  0^fi8v—  C0>  0 

•>C2—  UmU  0> 
CH  0  0(0  CO£- 

O  •  Ui0C  'Oivv 
V  •C0UI£0V‘D  —  £  U 

C20i.(OV  O0O0  *0r^ 

Mtt««0w  0  2(0-000) 

(uov  *>gE—  — (.0(0 

••>£3OOC0(.0£0O« 
gr>  0vav0«i>(.ci 
Viu—  E<  0—  000VCV2 
H'%,t.Oh>C<J>0v000(O 
O<<OC(O0(O2(O0  u  (.« 


0  0  V  >  • 

(-  0  3  —  V 
a—00  — 
vCC  — 
•  —  O  0  0 
a  —  c  c  u 

^  —  0  f- 

>  £  < 

(k  —  0 

O  0  —  i- 

lll“S 

V  —  Q  0 

**  a 

O  0  Z  C  0 
£  =  ..2^ 
2ca?a 

0  o  o  — 

>-  O  £  • 

0  V  0  0 


0  0 

tt  -  o  o 

^  V  4«  C 

>  0 

u.  >  C  V 

0-00 

.  *»>  — 

0  0  0—0 

3  £  0  a  0 

V  ♦*  0 

0  V  0 

V  «.  0  —  . 

0  O  V  £ 

0  £  U 

V  0  —  O  (. 
C  <.  0  -  « 
0  0  C  V  0 
0  0  C  0 
0  3  >  0  0 
(.  t.  (. 
O.V  0  V 

O  C  <  0 

i-fi  i 

V  g—  ^ 


S0  C  0  E 

DON 

0  V  i.  C  2  0 

>0  0  0 V  £ 

0  0  —  a  V 

>  0  L  0 

C  C  >  O  -  0  0 

0-0  £  E 

-  2  0  (.  V  0 

V  0  £  0  - 


jo  a  ‘  o  (o 

-  0  1  V  C 

0  <  0  O 

0  (.  2  L  - 
£  Ui  **  ** 
»-  *0  Z  0  0 
0tt-  — 
•  •  C  itJ  C  0 

-I  c  -  c 

V  0  (O  £  0 

►-  —*>2  t 
3  a<  <  f- 


O  Co 
V  o  «•  0  C 

O  0  O  s  - 
*• 

»  0  >  £ 

S55SS 

t  O  3  4  > 

0  C  0 

8*0^5 

cmmti 

0  C  0  £  O 
O  — 

Sv--^ 

2854*- 

a  0  w 

crs^» 

i*  .»5 

0  T>  X  0  0 

ft  0  c  g  0 
C  0  —  V 
0  C  C  » 

Z“5|| 

I-  a-  5  ♦< 


c  » 

0  V  £  0 

0  V  O 
•0^0 
0  V  C  I 
V  oa 

i.  p  ft 
o  0  C  -I 
-  o  o 


V  ^0 

C>  £  V  o  2  0 

0  0  3 

g-  o  c  o  0 
0£  o  V 

■r. 

Xft  —  0  V  0 
C  -I  X  (.  0  — 


P  X0 


—  V  £  r- 

Eo  • 

3  as 


c  3  as  0  u.  I 

0  L  C  K  r*  I 


—  V  —  I  0 

O.  0  »  0  N 

C  S25S 

—  O  0  Z  0 


(.0^0  ^ 
i  O  £  it 
{  V  C  I  £  X 

: 

ft  P  £  0  I.  0  - 

0  K  0  a£  £ 

(  «•  «i«  — 

>  0  •  —  0 

(.  P  V  0  0  C 

■ 

i  V  O  £  U  0  0 

i 

0  0  c  a  o  0 
a  0  0  0  V 

L  0  i  V  t.  0 
t  0  U  •  0  £ 

! 

V  0  0  3  — 

O  3  P  V  3 
X  C  0 

.  t  0  £ 

)  —  C  V  £  f  «• 

■  0  I  8.**  e  K  > 

•  0  0  0  0 

(  X  0  0X03 

J  0  >  P  -  0  0  2 

\  08"°  0  '*“  > 

I  Z  K  £  V  *•  i  — 


V  ••  V  0 

0  V 

0  C  C  0 

E  0  -  2 

Ul: 

Q.  0  0  0 

V  O.  3 

f  L  0 

ht- 

L  —  — 

8  V  et  8  a 
♦*  ow—  a 

t,  c  a 

3  (.  •£  ^ 

-«S8  S 

•CSV  9 
i  0  V  m 

!aS8 

0  C  L.  V 
X—  V  o 
0  p  0-^  O 
c  n  "x 

iSi-  8 

0  >  p 

00c  a 

V  p  o.  0  m 
o 

c  c  c 
-  o  o  • 
c  -  -  or* 
5  ‘V  4^  c  a 

i  p  0  0  —  m 
g  0  N  N  0  r( 

—  0  C  C0  — 
0  X  0  0  0  41 

>  a  a  o  I 


0  0  ft  V 
-  V  ft  c 
0  C  O  3 
Xi  U  z 


C  P  0 

0  0  C  0 

V  —  0  0  V 

0  V  V  £.  0 

2  u  c  V  p 

C  0  0 

c  3  1  <  0 

O  ft  0  c 

—  L  0  — 

V  —  0  0 

0  ••  3  —  V 

V  -  O'  a  c 

0  0  3  o 

ft  5  o 

0  0  O 

U  0  « 

0  *-  •  — 

a  X  —  X 

a  —  0  — 

•PC  •  p 

••  X  C  C  ft  c 

^  p  0  o  -  o 

I 

3  0  «  S  (.>< 


0  aS  g 

0  0-0 

—  ^  ^  ^ 

i 

ox  ft000a3fc. 

0  £  0  0  CM  — )  g 

H  ^93  CO 

p  (•  0  £  L  O  C 
-0  0iv00ug  c 

0(.  ftft  2>  £0  — 

CO  000  OC<0f 
00  Cftvo  —  — £ 

—  C  0CCV0  ftftv 
vg  4.0'^  vpov 

5  a  va  P00  cc. 
a  Pf-0X3O 

fa0C-(.£Oft 

£  M  o  -  au 

ft  V»-0EC00  P 
ft  •  Z-CO£LB0 
O0VV  £0-  acv 
U>C(.  •  VftPO—  o 

•-O3k50  £-00 

UI0— 0v£La0a> 
-0  C00a  00 


—  cc.  o—  P  009  O 
vvoaftc.  —  0aft  —  £0ftft 
00aafta0vc  £ftvo 


00aafta0vc  £ftvo  c 
E£  a<aC0—  03  O  CO 

C.0C-V  O0g0  oco  — 

0  0  COf^PCO  —  V 

ftP  —  X«0VO  —  £a—  V  —  V0 
C0VI.  00  00a(.0V0v 

-♦•00  —  C0>v-O£-i  — 

8E3**>C000  30PCU 
U  C  •1.03220  00 

0*-O0O0v  V  a£  ft  • 

O0ft3  acco0  cvmc— 
—  ac  movvcm— — pMCM 

c  M.ac  —  -ag3a  a 
£••  V  ocvx  ao  -  xa 
g  0  —  c  --0  0  aft  —  —  0  aw 
0000>VftVOO  0C9O  — 

K-uiiuf  c—  c>o£  —  ^ 

^C£g(U0C3£C.3  0££« 

3O>-0<M<OV0£ftO3K*-’ 


P  __  0  C  M  o 

C  P  O  0  0  - 

—  0  C  0  C  >0 

£—0  —  C.0PC 

♦-  -  -  ao  «-  0  0 
g  —  o  p  c  >  V 
0£0-5o-XP 
£0CPgE£O0C 
V  •  C  C  0  >  0 

P  ft  0  c  p  c  c  • 
cc—  00  —  X0 

-0030va  i.c 

0OVOCOVO 
0P  —  0—00  — 

E0C.  0—  c—  3V 
0V01.X0Ct.23 

—  gvo- «»«4ifCv 
£O0>0  ^c  —  — 

opoocpas  V 

COftOOC  OP0 

g  V  0  4/  (I  c  c 
0  P  0  C  c  0  - 
V£C0—  080 

c,0—  lOEto— 

0C  cc— 0VO0  ^a 

|5|||f|c||p£ 

—  C  —  V—  00  OVCZ 

00£0  cia30ca 
>20W'£  c2u0* 

880.  ■  0  0  O  C  >8  £ 
o  2  r.*'  c  £  0  _ 

0xci'?«'0— 0Pu8 

oa0O-C££0  c -< 


i 

0  £  P  — 

£•00  —  0 

V  c  -  0  —  6 

0-3  -  a 

u  -  fl  P  3 

O  £3  X  0 

ft  £  a  V  L  ft 

V  0  0  0  0M 

c  0  0  —  £  V  at- 

0£0g-cca 
—  1-32000'-’ 

£  5  -  c>  1. 

•  •  £  V  E 

0—  OvCfBC 
au  1.  0  O  £  C  C 

C  M  3  E  V  o  a 

0  I-  V  c  -  O 

C  o  3  t  0  £  C 

'—ft  £  0  0ft 

a  V  0  -  £ 

C  1.  0  0  £  V  C 

O  0  £  O  C  3  0 

V  ft  ft  —  0  — 

C  £  -  C  XV 

00(.av— 00 
0  0-3  -  E 

a  ft  —  0  V  £  c 

C  C  £  coo 


800 

0  V  U  C  I 

00  C  0  c 
0  C  C  90  I 
0X0B0  -L  J| 
0C£C(.  00  C 

0  2  V  -  0  Xft  0  • 

gi  Cft  V0E0C 
0  ftC0  — Lva 
(.PO0C  •(.  0C3 

£  0  —  0  0  X—  0  £ 

£  0  £  Xjt  £  0  L  I. 
au3  00VJC£  T 
uaa 

-0  £  ftOC  C4> 

C0>vt  0<OC 
C00  C"P£  -« 
0  0  0  C<0  ft  0  ft  t.  I 
-COw  CNOgOI 
£  L-*f—  £ftC 

_a2coic0ft_c 


'ft  — ocaof-ftM  0c 

0  g-0-0|pcv  'ft-g 

)0  Oft-£-C  MOC0  0 


jlili&llls! 

ft  C  L  CM  o  6  —  < 

BOO  C  0  •  P  >  J 
C  OftUOkftCPI 
-fC0MS  C00' 

£  0  2  V  W  0  0 

O  ft  (.  O  i  0  O  I 

M  0  c  (.  •  8  o  '  ' 

V  'ft— go'-'acu 


S* 


L  o  a  0 
o  —  o  > 
a  ft  u 


—  — ft  oogooa 

-fo  op-ftOLfta 

-£>3-Ca  -  CO 

£V—  OP0OP0  *0CM 
0  0-0  —  ixOftCM 

O-0_.E-o20c-- 
■^P^ac.0C3203i 

—  — ac0  —  £—  ftoa 

0Z£0—  Oft—  30Uft 


0  a  •  0 

£  K  X 

a  0  0 

IS  5* 

Z  O  ^  ' 

•080 

-i  £0  V 

ts^i 

3  0  «  O 


-  0  ft.  0  P  ft 

0  E  L  t-- 

>  (.  0  f  ft  —  0 

0  0  V  £m  >  .  o 

2  ft  0  0  c  X- 

C  (.  £  •  0  -  C 
ft  -  5  C  —  £ 

8-«8g?8 

2  0  O  £  £-  0  H 
0  O  O  V  ft  — 

0  c  «  -88-? 

-  0  ft  I.  P  0 

o  •  •  o  2 
c  0  a  gft  <  o 

&0  O  X  E  9'^ 
O  CM  —  >  0  ft  ft 

9lo%%^ii 

—  ft  M  V  0  — 

£0KCO3gO 

Koo  —  vft  oa 


O  C  0  0  Jt  0  O  3  0  « 

Ok- — oocojc- 

k—  30300001 
0£ftft>k—  E  3- 
ft  C£0Oft  pk- 

tf-2  op8o( 
p£f9  kOCft  ( 
8  ft  8  £  o  0  k  0  g  •  I 
£  a  ft  ft  a  8  0  ( 
o  a  0  x_  0  a  £  c 
-cc-pa  'Cxo; 

0—  0k8  0-0JC- 

xgiooxoccoi 

0O.-3£ftC3-« 

2  -  P  3  -  0  Q  < 

0COXC— — c 

0>0-a0-£00( 

k  0  ft  Oft  £  0 

02  Bk-ft-ft  i  -  I 

3  C0g-0ook( 
0—  £2kC  k0< 

£kC  630Pft£( 
O  O  C  0  £  £-  O  t 

£0-0900-0  X- 
3B£3E3ttft3£( 


r 


■» 


B-7 


u  V  S  ' 

T  C  o  u 


3 

a  >  g  H  i. 

^  9  m  m  90  >  O 

9  ^  i  t.  9  9  L,  a** 

2  •  C  I  •  C  U 
£^+«ot.«i  i-^a 
••  -•'••I.  l-H^. 

-  (.  •  •  5  « 

AAOMML'*' 
a  •  a 

»gi.CA-«>5ago 
‘909  orcoo 
I  **  **  a«-  ^  a  a  cm 
iaa-^oa4«c  ga  o 
(.c.-  uc<^ccc  >5 
■ao— a'^a&.oa4^««a 
laxtaccia-^  u«> 
-4^a««<^£-Hi«<aaiz 
c^Ckoc<-goa«^a 
- - 

fc. 

a 


*-  > 

9  —  * 

o  **  c 


I  a  a  u  o 

r  V  g  c  4*  —  c. 

*-3Sgia^ 


i+'f’^jcia  . 

aacLvi^oa  o 
2<Hco  caa  a*- 

Sa  a  »  c  0  c  L 

a>a-^-*-voaC4'in 
a(.lA  (££*^£9  « 

'SO 

a  3  a  a  a  (. 

.  _  o  «*. «-  ^  o 

^uS&cagaaj' 

«-S«>c»ai4'^«'U' 
occ^ . 


•  c  t  0  _  _ 
1  o  a  a  £ 
:  ■^  >♦#>*♦» 


^  ^  4^  4<  i  o 

_JgSi£«.€:. 

^004*04'  oaz 
■  ua«-iaa  c  ‘  -I 

5  ^  X  o  ?  ^  S  c  c  o 

paaa^a^'oP'-aa^ 

-u^oaa4«  f4<iitt 

ao  '-4'oaaco.c  go 
>.-aoc<.i£«.---o  Oio 
a  £C^3L4<aLC«>** 
<.4'£  00  4^o«>a< 

DO  uaa*.gca>>i 
:«.va£accoacao 
'wOHH  L<<-  aoT)  ao^ 


P 

g  u 

••  B«A 

111 

>  -  a 

£  O 
.jjfl. 

0  4*.- 

4*  a 
a  g  > 

>  c  a 

a  a  2 

L  C  •> 

a  a  a. 

4*  E  tt 
“20 
.£  O 

S  f 

U  £ 


;  csn  s 

a  M-  £  a  g  a 
aoucou^c 

SD?n«^|- 

a  a  c.  a  • 
a c  4*  2  4 

ll|55Sft 

u  a  u  g  V  o 

a  a  o  £  a  a  a 

4*£  .  4.  ao£ 

4.  a  3  a  4*  g 

iaSSSlc! 

a  c  4*  o  £  *•'  c 


.!  a 


C  &.  >^4*  <-  4» 

a  3  a  a  g  cl 

cgoSaaiL 
a  -  3  L  a  >  a 

aL4*a4.£.-a 

£  a  a  a  g  o  3 
v  4<  i  a3  f  > 

_  CLc.-ocg 

330  a30--  -.-C 

gr  o  9  o  <*■  jy  ^  a 
B  c  u  c  3  a  L 

00 

o£  'Eo  OLuaa 
4.  •^.a  Lcaa3a 

?(n‘*-xca<.>  a  a 
a  -004*  -iAV  £ 
-c*a  4*3. a  ca 
a-  oua«i-gaa  a 
ao  -0B4*ua  c 
cc..a  occ->o 
aa«/>>fou  —  aoa— 

Z4*Ma£L  a3>.C4. 

a»-L4*aCLgo  —  - 
•  Ltta  L4.0  cc 
^g34*a4*aaaac  — 
Hguc—aocc<La«. 
»-  c*vO££  g  oo  —  — a 
33<2»-4.  i  uu— ao 


a  u 

0  4*  —  — 

—  44  a  a  4* 

a  4*  0  0 

i*Ek?u, 

HbrisL 

g  o>iLaa>go 
ca-aia  oa£ 
oca  >>  x»c 
oa>Eg  LLO  L 

4*4*a»aa  4*a 

a  4*  4*Laaoco 

L^av30£-aa4* 

a  a  >«  0  g-  »-  £  -«i 

4*-  a  f-  4*  O  c 

—  a  aoa>  l>.o 

—  ocao4*aca>- 

—  00  a  -  04. 

00004*  4*04*  ‘C 

L£E  COOStAO 
fe*rt?caa3g-L 


4*  2  0  «- 

c  a  a  o 

o  £  g  »• 

-  »  -  o  >  o 

4*  3  L  4* 

ga^z: 

L  4.  a  4*  —  3 

o  a  o  £ 

—  a  »  a  a  a 

£  a  a  — 


a 

I 


0 

a  4*  4*  •  a 
N  c  4*  a  — 

llilit 


o  —  an 
L  a  oo 
a  c  L 
<  o  o.  r. 

—  n 
L  4*  a  « 
a  a  (.  CM 

3  c  a - 
«-  L  £  « 

a  a» 

•  van 
4*  c  0  2 
a  44  E 

C  4*  ui 

—  c  <  Ul 

*  4-  (/) 

aw 

O  U) 
o  n 


^  —  a 


!|D||£ 


4-  g  -  a  -  a 

of.  >  2. 

» 


£ 

4*  »  3  0  ^  0  0  4* 

L  —  a  a  a  4* 
aoa  —  -  g  4*c 

L4*Sca4*aBa4>> 
000-  C  LO- 

ai  -oa-4*aao 
94*  a  a  >  c  >  L  > 

^  .•£85l2ns£ 

a  a—  K  0  3 
a5o  —  —  aa  a. 

4.&CO-  -4*00-0 

a— aoaaoca  O 

-CE  gaa->^ 
a£  aE--»E-g- 
cuL  oao  gs- 

oacaL— L«»>a4*  — 
04*44.  —  vLOoaax 


8 

a  V. 

E8 


O  » 
—  r- 
4*  o> 

a  CD 
E  n 

L  — 

o  < 

I 

c  o 


-  .  o  c 
‘  o 

^  -  L  C 

0  33Ea4.£a 

4.  4.  L  B  U  4*  i 

a  a4.0L—  ao 

V  ovoLa-o 
a  a  -  c  4.  4*  a  2'^ 

-L-caJro^ 
c  £4*  --oao 

0  4*  a  4.  o  a  "v 

-  -  —  0  L  o  >*lf) 

4*  c  L  o  o  a  L  an 

Lzasaa  ag 

0  aao— a  c  — 

«.  .oa.o£ag£- 
c— a  — Nooa 

443L  -  aa  —  4*ar- 
a34*£CL04.ir> 

--  0.  ai/)  4*  o  5  I  - 

^  a£  o  £  < 

K  •aaaa4*4.ai 
t-4.£jCLa33-0 
3(AL>4*aLaac< 


a  jc 

a  0 
a  c 
£  a 


a  a.  >.2  • 

4*  i  C  CM 

0x0 

o.  a  o 

<  .•  a  —  fi. 

s  a  c  - 
a  4*  3 
—  o  a 
a  u  a  u  — 

!  *t> 

**  af  a  c  a 

“  a  < 


a  C9  —  o 
g  .j  10  L 

L  •  I  O 

a  2  a*-  4> 

X  2  L  O 
O  <  3  JC 
«-  S  £  -  O 
I-  C  -  O 

■  ■  a:  a  0  £ 
—  c  g 

•-  I  4.  3  C 

a  0  a 

3  <  ^  u  z 


— 
o  - 
»/>  — 

2i:s 

4*  a 
E  u 

a  a  a 

C  4*  L 
—  a  4* 
E  > 

L  V)  C 


g  V 

a  ^  a 
£  3  I 
4*  —  a 

g  S  - 

i-iS 

4.  a  a  4* 
o  a  —  L 


o  a 
o  o 

4*  C  4» 

0 

a  —  •— * 

0  4*  — 

a  w 

£  4* 

VV)  •• 

a  «- 

«8° 
a  fi 
4*  a  L 
C  I/I  0 

a  «• 

^  ^  2 


n  X  c  a  a  3 

-|g?£E 

•*  C  3  2  «> 

4.  —  o  a  a 
L  o  a  c 
0  a  g  a  a  o 
g  a  LO  — 

0  0  0  0  0  4* 

L  L  u  a  a  a 
u  L  3  E 


a  a  a 

-  w  a  a  c 

-  a  £  a  — 

£  g  £  3 

a  c  4*  a  a 

a  a  4.  c  4* 

0  ••  a  a  a 

a  -  0  g  o  g 

L  a 

4*  4*  a  «  a  o 
a  £  0  4* 

aaa>- 
3  £  a  a  4*  g 

a  L  L  L  3 


s  i 


S  a  a2 
4*  4*  L  E 


a  4-  -  0  1 

c  a  a 
—  4*  — 

c 

M* 


a** 


*  ^  Vi  £ 

tt  in  o  4* 

Q  a  VI 
a  VI  VI  c 
L  .  c  in  a  o 
3  c  o  w£  a 

4*  o  4*  4*  £  £ 

g  -  a  E  4*  4* 
a  4*  c  a  4-  o 
4*  —  —  4*  o  L  £ 

—  c  4*  a  o 

£  -  c  >  c  4-  . 

0  4-3  mo  a 
L  a  I  —  a  L 
a  g  a  4*  4*  a 
'-V  •  4*  —  c  X 

a  a  -  0  0  0  4* 

-  4*  o  o  -  i 

a  c  V  4-0  0 

i-f.  i«£' 

liU 

13  4*  ♦*  L 

•  -  --I  a 

-  _  _  a  a  L 

4*  fc  o  a  —  y  a 
a  L  o  c  i 
>—  4*  a  >  a  g 
a  a  a  ag  e  l 

c  <  m  3  L  a 
a  o  4*  o  £ 

4*  -  a  a  m  4. 
a  4*  a  £  L  4. 
go-  4*  a  a  o 
c  a  La 

c  3  3  c  3  4* 

0  tt.  0  —  4*  >  a 

—  o  o  a  a 

4*  jc  a  jc 

a  ■•  —  0  4*  a 

■I  T  g  £ 


SB*  a 

a  c 


iU 


a  a  a _ 

— .  V  4*  4*  a  a 
n  c  c  c  L 

4*LLL-£ 

L  —  —  —  4*<o  ••>'£->4*m 

BBBBa*i!l%z  Ksaoaaila 

aaaa£a35w  »-4*oa£C3£ 

tLLL4*g‘"*“ - - - - 


I  a  — 

I  c  >  « 
j  0  a  r- 

':^VS 

■*!SS 


m  0  .  *  £  ?  4* 
jc  c  4.  o  a 
a  a  c  —  —  L  a 
oao— a«  —  — 
0  4-0  a  -  V.  a 
a  g  o  4*  a  c  m 
VI  2  —-00 

•  c  a  —  m 

•  £  -  >4*  m 


3a<jaK<4.Ha 


3  C  ? 
E  >4.  a  a 

a  g  a 

g  4*  3  a  a  a 
c  *  a  V  3  g  a 
a  a  g  >.  a  —  u 
4*  a  a  a  a  >  - 
aav4*  v>o> 
gLcaacoLL 
aac£aLaa 
gEi-4*2a  a 
-  0  a  g  5*  > 

a4*LLLaofc 
aa— ooaafo 
£  3  00^  £ 

u  g  g  3  0  4* 

a  a  a  a  v  a 
rcLo-  gas 

VO  a  L  —  £  L 

-aaaagvo 

av£gx)£&  4- 

«a  V-  V  £  c  c 
V  >  a  o  a  —  — 
—  m  4-  o 

>  OLOLaao 

oa  aaovar 
L  g  a  4.  c  —  V 
a  a  a  o  V  9  ** 
aovagi  —  4. 
vma  aca>o 
a  La  a  L  — 
£L3aa  — va 
voaxvm2wg 
4.  c  o  g  a  o 
>  >.  a  m  a  g 
aoLvimLca 
g  a  a  —  a  o 
agfi>Laa— a 
a-  -  -  -  £  a  V  £ 
a  >  L  V  2*^  i  a 

L  a  u  g  £  V 
ca  aa4.4-m«. 
-  a  a  L  o  o 

£  V  >0 

gvi-cgcaoc 

S|  .|S2*&£ 

^oa-gcxLu 

BS:53Cvig 


E  a 

•  h 

L  jc  a 
aao 
o  a  - 
L  V  o 
a  a 
V  0  c 
c  c  V  o 
0  a  - 


E  a  c 

a  a  a  £  - 

V  £  >.  V  V  o 

a  V  a  a  a  o 

-  JC  >.  L  >  •  'v 

tL  m  0  O  -  V  CM 

0  g  4.  V  c  - 
4-  c  a  E  u  a 
o  avvoaiio 
g  L  a  a  • 
a  4.  a  r 


L  a  o  1 


—  |-V4.>cc3£a« 
vaaoo-aoovaa> 
ao3v  LE-uo  auj 
V  —  ♦*i/)**gav  £l  k 

mag  c  lob  ox** 
>aaac-v£g****< 

a  oaoaaaaoaan 

aaom— “*• 

•~  ♦*  *■ 
c  a  ^ 


L  a 


E  a 


2  ivS«i?' 

V  a  o  >  a  o  < 


OL-au- 
a  a  a  o  -  z 
-  a  g  >  V  * 
a  m  0  a  g  g  V 

.  .oc^olS 
i.  0—  -£aa  a 

C—  CVCMOVV  OL  00 
-2©''— 4*0  Lxaan 

g-g  ccLo  |vBD 
gave. -030  vocr- 
auaa— EvLcavar- 
g  g  ma  oaai- 
aa— a'-'L4.a— gsa-- 
a£v£m  —  ovv  g  0  L» 
Cv  —  4*102  C—  O  —  CM 

♦*  wgvacgg2' 

a4.CL  acE-  a  g- 
£oaoaLaa4.aoa 
04.1  ELic-Lm 
-a  -  a  -  g-  g  -  >  < 

—  V  avao2  vaa2 

—  —  —Lac—  g—  a  a 
X3-3>>oaaa^L  xco 

awvm  —  >  Lwf  c  on 
va  o  VO  go  — « 
aL-icygago--r- 
£  avoc  ye  var- 
♦'LX--3— caaa  - 
oocvvna  £voin 
a-»-oa  wE*-»-mvar- 

EO— LEV  L—  ua 

iSOOLO-'Om  -  ELiO 
V  v_o  mvomgaacM 
aa  gvvMLmMogmz 
>>£accamfmmg&.<a 
a  v  a  a  •-  a  m  aw  m  m  o  2  a 


.  .  g 
aao 
g  Q  g  ^ 
a  2  0 

Sci28 

£  o  u  v  a 
<  —  o  a  a 

L  V's.n  £ 

9  an 


a 

>.  V 

a  a  c  • 
X  a  > 
a  a  u  - 

V  •»  a 

-  a  a  c  3 
a  a  a  g  0  o 
o  V  o  —  a 
o  o  V  «  ♦*  c 
-EC  a  a 

a  a  o  E  V 
a  L  a  V  L  — 

-  0  3 

V  0  —  a  V  S 
o  V  —  —  c  - 

•  £*-■•) 
c  a  V  1 
o  a  c  —  a 

—  a  —  a  a  a 

V  u  £  g  a 
age  -  £ 
L  a  a  —  c 
a  -  £  a 
a  £  £  —  u  V 

V  V  a  X  a  a 
c  —  3  K  g 

-  X  0  E 

L  a  a  — 

a  E  £  V  a  tt 
£  a  4-  a  c  »- 
♦*  ♦*  >  a  w 

a  •  a  V 
a  >  a  a  a 

V  a  a  a  o  V 
a  a  £  L 
L  V  a  V  0  o 

V  c  £  £  a 

a  0  V  V  a 
c  I  a  0  .  a 
0  a  V  g 

I aa  oc- 
i  a  g  L  a  a 
g  c  a  g 

z^i’n 

—  c  a  •  a  g 
X  0  a  L  V  a 

—  Loa»- 

V  V  a  a  u 

88l:-o 

—  o  a 

o  —  £oy»- 

L  o  -  L  0  O 
a  u  £  a—  w 


B-10 


3-11 


